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Figure 3—The layers of protocol 


tions are often provided (e.g., saving a transcript of a 
session in a local file, sending a file in place of user- 
typed input, reporting whether various HOSTs are or 
have been up). 

In the serving HOST it is desirable that a process 
controlled over the ARPANET behave as it would if 
controlled locally. The cleanest way to achieve this 
goal is to generalize the terminal con'trol portion (TCP) 
of the operating system to accept ARPANET terminal 
interaction. It is unpleasant to modify any portion of 
a working computer system and modification could be 
avoided if it were possible to use a non-supervisor 
process (e.g., "server-TELNET” or "LOGGER”) to 
perform the job creation, login, terminal input-output, 
interrupt, and logout functions in exactly the same way 



Figure 4—Data flow for remote interactive use 
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Figure 6—Data Bow scheme for server 


as a direct console user. Prior to the development of the 
ARPANET, no operating system provided these func¬ 
tions to non-supervisor processes in anywhere near the 
required completeness. Some systems have since been 
modified to support this generalized job control scheme. 
See Figures 5 and 6. 

Efforts to standardize communications in the TEL¬ 


NET protocol focused on four issues: character set, 
echoing, establishing connections, and attention 
handling. 

The chosen character set is 7-bit ASCII in 8-bit 
fields with the high-order bit off. Codes with the high- 
order bit on are reserved for TELNET control func¬ 
tions. Two such TELNET coptrol function codes are 
the “long-space” which stands for the 200 millisecond 
space generated by the teletype BREAK button, and 
the synchronization character (SYNCH) discussed be¬ 
low in conjunction with the purpose of the TELNET 
interrupt signal. 

Much controversy existed regarding echoing. The 
basic problem is that some systems expect to echo, 
while some terminals always echo locally. A set of con¬ 
ventions and signals was developed to control which 
side of a TELNET connection should echo. In practice, 
those systems which echo have been modified to include 
provision for locally echoing terminals. This is a non¬ 
trivial change affecting many parts of a serving HOST. 
For example, normally echoing server HOSTs do not 
echo passwords so as to help maintain their security. 
Terminals which echo locally defeat this strategy, how- 



LOGGER must be a background service program capable of initiating jobs 


Figure 6—Alternate data flow scheme for a server 
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Figure 7—Data flow and processing of the character 
input stream 


ever, and some other protection scheme is necessary. 
Covering the password with noise characters is the 
usual solution. 

The HOST-HOST protocol provides a large number 
of sockets for each HOST, but carefully refrains from 
specifying which ones arc to be used for what. To estab¬ 
lish communication between a user-TELNET and a 
server-TELNET some convention is required. The 
Initial Connection Protocol (ICP) 10 is used: 

1. Connection is initiated from a uscr-TELNET’s 
receive socket to a serving HOST’S socket 1 
(a send socket). 

2. When the initial connection is established, the 
serving HOST sends a generated socket number 
and closes the connection. This socket number 
identifies an adjacent socket pair at the serving 
HOST through which the user-TELNET can 
communicate with a server-TELNET. 

3. TELNET connections are then initiated be¬ 
tween the now specified pairs of sockets. Two 
connections are used to provide bi-directional 
communication. 

Note that socket 1 at the serving HOST is in use only 
long enough to send another socket number with which 
to make the actual service connections. 

One of the functions performed by a terminal control 
program within an operating system is the scanning of 
an input stream for attention characters intended to 
stop an errant process and to return control to the 
executive. Terminal control programs which buffer in¬ 
put sometimes run out of space. When tins happens to 
a local terminal’s input stream, n “bell” or a question 


mark is echoed and the overflow character discarded, 
after ehcckmtj to see if it is the attention character. See 
Figure 7. This strategy works well in practice, but it 
depends rather strongly on the intelligence of the human 
user, the invariant time delay in the input transmission 
system, and a lack of buffering between type-in and at¬ 
tention checking. None of these ebullitions exists for 
interactive traffic over the net: The serving HOST can¬ 
not control the speed (except to slow it down) or the 
buffering within the using HOST, nor can it even know 
whether a human user is supplying the input. It is thus 
necessary that the terminal control program or server- 
TELNET not, in general, discard characters from a net¬ 
work input stream; instead it must suspend its accept¬ 
ance of characters via the HOST-HOST flow control 
mechanism. Since a HOST may only send messages 
when there is room at the destination, the responsibility 
for dealing with too much input is thus transferred back 
to the using HOST. This scheme assures that no charac¬ 
ters accepted by the using HOST are inadvertently lost. 
However, if the process in the serving HOST stops ac¬ 
cepting input, the pipeline of buffers between the user- 
TELNET and remote process will fill up so that atten¬ 
tion characters cannot get through to the serving 
executive. In the TELNET protocol, the solution to 
this problem calls for the user-TELNET to send, on 
request, a HOST-HOST interrupt signal forcing the 
server-TELNET to switch input modes to process net¬ 
work input for attention characters. The server- 
TELNET is required to scan for attention characters 
in its network input, even if some input must be dis¬ 
carded while doing so. The effect of the interrupt signal 
to a server-TELNET from its user is to cause the buf¬ 
fers between them to be emptied for the priority pro¬ 
cessing of attention characters. 

To flip an attention scanning server-TELNET back 
into its normal mode, a special TELNET synchroniza¬ 
tion character (SYNCH) is defined. When the server- 
TELNET encounters this character, it returns to the 
strategy of accepting terminal input only as buffer 
space permits. There is a possible race condition if the 
SYNCH character arrives before the HOST-HOST 
interrupt signal, but the race is handled by keeping 
a count of SYNCHs without matching signals. Note 
that attention characters are HOST specific and may 
be any of 129 characters—128 ASCII plus “long 
space’’—while SYNCH is a TELNET control character 
recognized by all servcr-TELNETs. It would not do 
to use the HOST-HOST signal alone in place of the 
signal-SYNCH combination in attention processing, 
because the position of the SYNCH character in the 
TELNET input stream is required to determine where 
attention processing ends and where normal inode input 
processing begins. 
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FILE TRANSFER 

When viewing the ARPANET as a distributed 
computer operating system, one initial question is that 
of how to construct a distributed file system. Although 
it is constructive to entertain speculation on how the 
ultimate, automated distributed file system might look, 
one important first step is to provide for the simplest 
kinds of explicit file transfer to support early sub¬ 
stantive use. 

During and immediately after the construction of the 
ARPANET user-level process interface, several ad hoc 
file transfer mechanisms developed to provide support 
for initial use. These mechanisms took two forms: (1) 
use of the TELNET data paths for text file transfer and 
(2) use of raw byte-stream communication between 
compatible systems. 

By adding two simple features to the user-TELNET, 

. text file transfer became an immediate reality. By 
adding a “SCRIPT” feature to user-TELNETS 
whereby all text typed on the user’s console can be 
directed to a file on the user’s local file system, a user 
need only request of a remote HOST that a particular 
text file be typed on his console to get that file trans¬ 
ferred to his local file system. By adding a “SEND- 
FILE” feature to a user-TELNET whereby the con¬ 
tents of a text file can be substituted for console type-in, 
a user need only start a remote system’s editor as if to 
enter new text and then send his local file as type-in 
to get it transferred to the remote file system. Though 
crude, both of these mechanisms have been used with 
much success in getting real work done. 

Between two identical systems it has been a simple 
matter to produce programs at two ends of a connection 
to copy raw bits from one file system to another. This 
mechanism has also served well in the absence of a more 
general and powerful file transfer system. 

Ways in which these early ad hoc file transfer mech¬ 
anisms are deficient are that (1) they require explicit 
and often elaborate user intervention and (2) they de¬ 
pend a great deal on the compatibility of the file sys¬ 
tems involved. There is an on-going effort to construct 
a File Transfer Protocol (FTP) 11 ' 12 worthy of wide 
implementation which will make it possible to exchange 
structured sequential files among widely differing file 
systems with a minimum (if any) explicit user inter¬ 
vention. In short, the file transfer protocol being de¬ 
veloped provides for the connection of a file transfer 
user process (“uscr-FTP”) and file transfer server 
process (“server-FTP”) according to the Initial Con¬ 
nection Protocol discussed above. See Figure S. A user 
will be able to request that specific file manipulation 
operations be performed on his behalf. The File Trans¬ 
fer Protocol will support file operations including (1) 



Figure 8—Data flow for file transfer 


list remote directory, (2) send local file, (3) retrieve re¬ 
mote file, (4) rename remote file, and (5) delete remote 
file. 

It is the intention of the protocol designers to regu¬ 
larize the protocol so that file transfer commands can 
be exchanged by consoles file transfer jobs engaged in 
such exotic activities as automatic back-up and dy¬ 
namic file migration. The transfers envisioned will be 
accompanied with a Data Transfer Protocol (DTP) 11 
rich enough to preserve sequential file structure and in 
a general enough way to permit data to flow between 
different file systems. 

USING THE ARPANET FOR REMOTE 

JOB ENTRY 

A very important use of the ARPANET is to give a 
wide community of users access to specialized facilities. 
One type of facility of interest is that of a very powerful 
number-cruncher. Users in the distributed ARPANET 
community need to have access to powerful machines 
for compute-intensive applications and the mode of 
operation most suited to these uses has been batch 
Remote Job Entry (RJE). Typically, a user will generate 
a “deck” for submission to a batch system. See Figure 9. 
He expects to wait for a period on the order of tens of 
minutes or hours for that “deck” to be processed, and 
then to receive the usually voluminous output thereby 
generated. See Figure 10. 

As in the case of file transfer, there arc a few useful 
ad hoc ARPANET RJE protocols. A standard RJE 
protocol is being developed to provide for job sub¬ 
mission to a number of facilities in the ARPANET. 
This protocol is being constructed using the TELNET 
and File Transfer protocols. A scenario which sketches 
how the protocol provides the RJE in the simplest, 
most explicit way is as follows: 

Vie an ARPANET RJE process, a user connects his 
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Figure 9—Submission of RJE input 


terminal to an RJE server process at the HOST to 
which he intends to submit his job “deck.” Through 
a short dialogue, he establishes the source of his input 


and initiates its transfer using the File Transfer Proto¬ 
col. At some later time, the user reconnects to the ap¬ 
propriate RJE server and makes an inquiry on the 



Figure 10—Retrieval of RJE output 
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status of his job. When notified that his input has been 
processed, he then issues commands to the serving 
HOST to transfer his output back. 

We can of course imagine more automatic ways of 
achieving these same functions. A user might need only 
type a job submission command to his local system. 
Automatically and invisibly, then, the local system 
would connect and converse with the specified RJE 
server causing the desired output to later appear in the 
users file area or perhaps on a local line printer. The 
intention is to design the RJE protocol so that the ex¬ 
plicit use can start immediately and the more automatic 
RJE systems can be built as desired. 

OTHER PROTOCOLS AND CONCLUSIONS 

One of the more difficult problems in utilizing a net¬ 
work of diverse computers and operating systems is that 
of dealing with incompatible data streams. Computers 
and their language processors have many ways of 
representing data. To make use of different computers 
it is necessary to (1) produce a mediation scheme for 
each incompatibility or (2) produce a standard repre¬ 
sentation. There are many strong arguments for a 
standard representation, but it has been hypothesized 
that if there were a simple way of expressing a limited 
set of transformations on data streams, that a large 
number of incompatibilities could be resolved and a 
great deal of computer-computer cooperation expedited. 

The bulk of protocol work is being done with the 
invention of standard representations. The TELNET 
protocol, as discussed, is founded on the notion of a 
standard terminal called the Network Virtual Terminal 
(NVT). The File Transfer Protocol is working toward 
a standard sequential file (a Network Virtual File?). 
So it is also with less advanced protocol work in graphics 
and data management. 

There is one experiment which is taking the trans¬ 
formational approach to dealing with incompatibilities. 
The Data Reconfiguration Service (DRS) is to be 
generally available for mediating between incompatible 
stream configurations as directed by user-supplied 
transformations. 13 

ACKNOWLEDGMENTS 

Function-oriented protocols have been the principal 
concern of the ARPANET Network Working Group 
fNWG). A list of people who have contributed to the 
development of the protocols discussed would include, 
Robert Braden, Howard Brodie, Abhay Bhushan, 


Steve Carr, Vint Cerf, Will Crowther, Eric Harslem, 
Peggy Karp, Charles Kline, Douglas McKay, Alex 
McKenzie, John Melvin, Ed Meyer, Jim Michener, 
Tom O’Sullivan, Mike Padlipsky, Arie Shoshani, Bob 
Sundberg, A1 Vezza, Dave Walden, Jim White, and 
Steve Wolfe. We would like to acknowledge the contri¬ 
bution of these researchers and others, in the ARPA 
Network Working Group, without assigning any re¬ 
sponsibility for the content of this paper. 

REFERENCES 

1 L G ROBERTS B D WESSLER 

Computer network development to achieve resource sharing 
AFIPS Conference Proceedings May 1970 

2 F E HEART et al 

The interface menage processor for the ARPA 
computer network 

AFIPS Conference Proceedings May 1970 

3 L KLEINROCK 

Analytic and simulation methods in computer 
network design 

AFIPS Conference Proceedings May 1970 

4 H FRANK I T FRISCH W CHOU 
Topological considerations in the design of the ARPA 
Computer network 

AFIPS Conference Proceedings May 1970 

5 Specifications for the interconnection of a Host and an IMP 
Bolt Beranek and Newman Inc Report No 1822 
February 1971 

6 C S CARR S D CROCKER V G CERF 
HOST-HOST communication protocol in the 
ARPA Network 

AFIPS Conference Proceedings May 1970 

7 HOST/HOST protocol for the ARPA Network 
ARPA Network Information Center #7147 

8 T O’SULLIVAN et al 
TELNET protocol 

ARPA Network Working Group Request For 
Comments (RFC) #158 ARPA Network Information 
Center (NIC) #6768 May 1971 

9 S M ORNSTEIN et al 

The Terminal IMP for the ARPA Computer Network 
AFIPS Conference Proceedings May 1972 

10 J B POSTEL 

Official initial connection protocol 

ARPA Network Working Group Document #2 NIC 

#7101 June 1971 

11 A K BHUSHAN et al 
The data transfer protocol 

RFC #264 NIC #7812 November 1971 

12 A K BHUSHAN et al 
The file transfer protocol 

RFC #265 NIC #7813 November 1971 

13 R ANDERSON et al 

The data reconfiguration service—An experiment 
in adaptable process j process communication 
The ACM/IEEE Second Symposium On Problems In 
The Optimization Of Data Communications Systems 
October 1971 


TIP USER DIALOGUE 


E 

HELLO 

@ HOST 23 

@ LOGIN 

TR OPEN 

LOG ON * 


LOG OFF * 

@ CLOSE 


TR CLOSED 




McROSS—A multi-computer programming 


system* 


by ROBERT H. THOMAS and D. AUSTIN HENDERSON 

Bolt, Beranek and Newman, Inc. 

Cambridge, Massachusetts 


INTRODUCTION 

This paper describes an experimental "distributed” 
programming system which makes it possible to create 
multi-computer programs and to run them on com¬ 
puters connected by the ARPA computer network 
(ARPANET). 1 The programming system, which is 
called McROSS (for Multi-Computer Route Oriented 
Simulation System), is an extension of a single- 
computer simulation system for modelling air traffic 
situations 2 developed by Bolt, Beranek and Newman, 
Inc. (BBN) as a tool for air traffic control research. 
The McROSS system provides two basic capabilities. 
One is the ability to program air traffic simulations 
composed cf a number of “parts” which run in geo¬ 
graphically separated computers, the distributed parts 
forming the nodes of a “simulator network.” The 
second is the ability of such a simulator network to 
permit programs running at arbitrary sites in the 
ARPANET to “attach” to particular nodes in it for the 
purpose of remotely monitoring or controlling the 
node’s operation. 

The McROSS distributed programming system is 
unique in several ways: 

(a) Use of McROSS generates inter-computer traffic 
in which a group of programs are engaged in 
substantive conversation. There is relatively 
little previous experience with such inter¬ 
computer, program-to-pfogram conversations. 

(b) The component nodes of a simulator network 
are not bound to particular ARPANET sites 
until simulation “run time.” Thus on different 
runs the same distributed program can be dis¬ 
tributed in different ways over the ARPANET. 
For example, in one run all the nodes of a simu- 
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lator network might be run at BBN and on the 
next some might be run at BBN, others at 
RAND and still others at the University of 
Utah. This mode, of using the ARPANET is 
significantly different from the normal one in 
which programs are bound to particular net¬ 
work sites at program composition time. (The 
only constraint on the binding of nodes to 
ARPANET sites is the requirement that each 
node run on a PDP-10 under the TENEX 
operating system. 1 ) 

(c) The responsibilities of a node in a simulator net¬ 
work can be conveniently partitioned into sub- 
tasks which can be performed more or less 
independently of one another. The McROSS 
implementation mirrors this partitioning. Func¬ 
tions performed at the nodes are realized by 
groups of loosely connected, concurrently evolv¬ 
ing processes. 

The distributed simulation system represents an 
initial step in an on-going research program which is 
investigating techniques to make it easy to create, run 
and debug computations involving the coordinated be¬ 
havior of many computers. The McROSS system is 
intended to serve both as an experimental vehicle for 
studying problems related to distributed computation 
and as a tool for air traffic control research. Its two 
goals arc well matched. A satisfactory solution to the 
nation’s air traffic problems is likely to include a net¬ 
work of airborne and ground based computers working 
together on a single distributed computation: the 
scheduling and control of aircraft maneuvers. Thus, the 
air traffic control problem is a rich source of interesting 
problems in partitioned computation which can be used 
to measure the usefulness of the distributed computa¬ 
tional techniques developed. 

This paper is a report on one phase of a continuing 
research program. The intent is to describe interesting 
aspects of an experimental distributed programming 
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system and to share our experience with others con¬ 
sidering the construction of such systems. The paper 
docs no more than hint at how the McROSS system 
can be used in air traffic control research. 

The next section provides background useful for 
subsequent discussion of the distributed programming 
system. After than, the McROSS system is described, 
first in terms of the facilities it provides for creating 
distributed simulations, and then in terms of interesting 
aspects of its implementation. The paper closes with a 
discussion of some of the issues brought into focus by 
the experience of implementing and using a distributed 
programming system. 

COMPONENTS OF THE McROSS SYSTEM 

The main components of the distributed program¬ 
ming system are: 

1. The ARP A computer network; 

2. The TENEX operating system for the PDP-10; 
anc. 

3. A simulation programming system known as 
ROSS (for Route Oriented Simulation System). 

These components became operational within 6-10 
months of one another, at which time it became feasible 
to implement the distributed programming system 
being described. 

The ARPANET 

The ARPA computer network 1 is a set of auton¬ 
omous computer systems (hosts) which are intercon¬ 
nected to permit resource sharing between any pair of 
them. The goal of the ARPANETTs for each host to 
make its resources accessible from other hosts in the net 
thereby permitting persons or programs residing at one 
netwmrk site to use data and programs that reside and 
run at any other. Each host interfaces with the network 
through an Interface Message Processor (IMP), 3 a 
small dedicated general-purpose computer. The IMPs, 
which reside at the various ARPANET sites, are con¬ 
nected by 50 kilobit common carrier lines and arc 
programmed to implement, a store and forward com¬ 
munication network. In passing from host A to host B 
a message passes from host A to IMP A, through the 
IMP communication network from IMP A to IMP B 
(in general passing through a number of intermediate 
IMPS) and finally from IMP B to host B. 

At each host there is a Network Control Program 
(NCP) whose function is to provide an interface be¬ 


tween the network and processes within the host. The 
NCPs use the IMP store and forward network to pro¬ 
vide processes with a connection switching network. 
Thus, they enable processes in different hosts to estab¬ 
lish connections with one another and to exchange 
information without directly concerning themselves 
with details of network implementation such as the 
way in which hosts are connected to and communicate 
with IMPs. 

Information can flow in only one direction on an 
ARPANET connection. Thus, before two processes are 
able to engage in a dialogue they must establish two 
such connections between them. A connection is com¬ 
pletely specified by its two ends which are called 
sockets. A network socket is itself uniquely specified by 
a host identifier and a socket identifier. The purpose of 
the socket identifier is to specify a process or group of 
processes within a host and a socket relative to that 
process or process group. Thus, it is useful to think of a 
socket as having a three component “socket name” of 
the form H.P.N. H is the “host” component which 
identifies an ARPANET host, P is the “process” com¬ 
ponent which identifies a process group within H and N 
is the “process-local” component which identifies a 
socket relative to P. In the sequel 

H,.P,.N,-*H,.P,.N, 

is used to denote a connection between sockets 
Hi.Pi.Ni and H 2 .Pj.Nj where —> indicates the direc¬ 
tion of information flow over the connection. 

For a connection to be established between two 
processes each must request that it be established. There 
are two common ways in which connections are es¬ 
tablished. In the first, the processes play symmetric 
roles. Each specifics, as part of a “request for connec¬ 
tion” (RFC), the socket name at its end of the con¬ 
nection, the socket name at the remote end of the 
connection and the direction of information flow. If the 
two RFCs “match” the connection is established. This 
connection procedure requires a priori agreement upon 
the sockets to be used for the connection. The second 
common connection procedure is used in situations in 
which one process wishes to provide some sort of service 
to other processes. The “serving” process establishes 
a listening socket within its host which is receptive to 
RFCs from any other process in the network and then 
“listens” for connection attempts. The serving process 
uses facilities provided by its NCP to detect the occur¬ 
rence of a connection attempt and to determine its 
source. When such an attempt is made the serving 
process can choose to accept or reject it. This connec¬ 
tion procedure requires that only one socket name, 
that of the serving process’s listening socket, be known 
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a priori. In the remainder of thin paper 

[h.p.n--]l 

and 

[H.P.N-]l 

are used to denote connections established in a listening 
state. 

The TENEX operating system 

TENEX‘ is a time sharing system for the DEC 
PDP-10 processor augmented with paging hardware 
developed at BBN. For purposes of this paper it is 
useful to describe TENEX in terms of the virtual 
processor it implements for each logged-in user (i.e., 
user time sharing job). 

The instruction repetoire of the TENEX virtual 
processor includes the PDP-10 instruction set with the 
exception of the direct I/O instructions. In addition, it 
inclui^o i-.-iUuctions which provide access to virtual 
processor capabilities implemented by the combination 
of the TENEX software and hardware. 

The TENEX virtual processor permits a user job to 
create a tree-structured hierarchy of processes. Such 
processes have independent memory spaces and com¬ 
putational power. At different stages in its lifetime a 
single user job may include different numbers of pro¬ 
cesses in various states of activity. Several mechanisms 
for interprocess communication arc provided by the 
TENEX virtual machine. Processes can interact by 
sharing memory, by interrupts and by direct process 
control (e.g., one process stops, modifies and restarts 
another which is “inferior” to it in the hierarchy). 

A memory space is provided by the virtual processor 
which is independent of the system configuration of 
core memory. Each process has a memory space of 
256K words which is divided "into 512 pages each of 
512 words. A process can specify read, write and execute 
protection for pages in its memory space as it secs fit. 

The virtual machine includes a file system which 
provides a mechanism for storing information on and 
retrieving it from external devices attached to TENEX. 
Processes refer to files using symbolic names, part of 
which identifies the particular device on which the 
file resides. The instruction set of the virtual machine 
includes operations for data transfer to and from files 
which a process can execute without explicitly arrang¬ 
ing for buffering. 

The NCP resident in TENEX makes the ARPANET 
appear to TENEX processes as an I/O device. The 
name of a “file” corresponding to a network connection 
includes the names of both the local socket and the 


remote socket which define the connection. A process 
requests TENEX to establish a connection by attempt¬ 
ing to open an appropriately named “network” file. 
The open attempt succeeds and the connection is es¬ 
tablished if and when another process issues a matching 
RFC. TENEX processes transmit and receive informa¬ 
tion over network connections by executing the normal 
file data transfer instructions. 

ROSS 

ROSS is a programming system for modelling air 
traffic situations. 2 It includes facilities for creating and 
running simulation experiments involving air traffic in 
an experimenter-defined airspace. The system is cur¬ 
rently being used in a number of air traffic control re¬ 
search projects. 

To create a simulation experiment the experimenter 
uses ROSS language forms to write a “program” which 
defines the geography of an airspace, the wind profile 
for the airspace, aircraft flight characteristics and a set 
of “route procedures.” A route procedure consists of a 
sequence of one or more “instructions” for monitoring 
or controlling the progress of an aircraft through the 
simulated airspace. Execution of a route procedure is 
the ROSS counterpart of pilot and/or controller actions. 
The “flight” of a simulated aircraft through the air¬ 
space is accomplished by the execution of a series of 
route procedures. ROSS includes "primitive” route 
procedures to “create” aircraft, “inject” them into and 
remove them from the simulated airspace as well as 
ones which cause aircraft to accelerate, turn, climb and 
descend. 

By compiling his program the experimenter creates 
a simulator for traffic in the airspace he has defined. 
To perform a simulation experiment the experimenter 
runs his program. The simulated airspace remains 
empty until the program is supplied with input which 
inject aircraft into the airspace and control their flight 
through it. Each input line the simulator receives is 
passed to an internal parsing mechanism which issues 
a call to an appropriate experimenter-defined or primi¬ 
tive route procedure. The program can accept input 
from an on-line terminal, a predefined “scenario” file, 
or both. Input lines from the scenario file art’ identical 
to ones from the on-line keyboard with the exception 
that scenario file input lines include a time, field which 
specifies the (simulated) time at which the program is 
to accept them. A user can manually "vector” aircraft 
through the airspace by supplying input at the on-line 
keyboard. 

A ROSS simulator can drive displays of the airspace 
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and, in addition, can generate output which can hr 
written into files, typed on the on-line keyboard or 
sent hack into the simulator input parser. 

THE McROSS PROGRAMMING SYSTEM 

The McROSS system provides the ability to define 
simulation experiments involving a number of airspaces 
or “simulation centers’’ which can be interconnected 
to form a simulator network. Adjacent centers in the 
simulator network are connected to one. another by way 
of the ARPANET. The components of a simulator net¬ 
work may run as user jobs distributed among different 
TENEXsoros different user jobs on the same TENEX. 
(As of January 1972 the ARPANET included five 
TENEX hosts.) 

Computational responsibility for performing a multi¬ 
computer McROSS simulation is truly distributed. 
For ex"mp!c, as an aircraft flies from one airspace into 
an adjacent one the responsibility for simulating its 
dynamics •’hifts from one computer to another. 

Goals 

The McROSS system was implemented to achieve 
the following goals: 

1. Autonomy of parts: 

Individual components of a McROSS network 
should be able to operate independently of one 
another (to the extent that each is independent 
of the others for traffic). Furthermore, no center 
should be able to cause another to malfunction. 
Autonomy of parts enables a multi-computer 
simulation to run when only some of its compo¬ 
nents are operational. Failure of a component 
center in a multi-center simulation results in 
degradation of the total simulation rather than 
forced termination of it. A beneficial side effect 
of autonomy is that individual centers can be 
partially debugged without running the entire 
simulator network. 

2. Deferral of process/processor binding: 

The binding of centers in a McROSS network to 
host computers in the ARPANET should be de¬ 
ferred until run time. This goal can be stated in 
more general terms. The program for a dis¬ 
tributed computation defines a logical configura¬ 
tion made up of abstract relations between the 
computation’s parts. A given execution of the 
program is accomplished by a particular physical 
configuration of computers. The two configura¬ 
tions, logical and physical, are by necessity re¬ 


lated. However, the programmer should have 
the option of specifying them separately. By 
deferring proccss/proccssor binding the con¬ 
figurations can be separately specified. As a 
result the programmer is free while composing 
his program to concentrate on the logical struc¬ 
ture he wants his program to define without 
concerning himseif with details of the physical 
structure on which it is to be run. 

3. Capability for dynamic reconfiguration: 

In the course of a simulation it should be pos¬ 
sible for adjacent centers to dynamically break 
and reestablish connections with one another. 
Furthermore, it should be possible for process/ 
processor binding to be changed during a simula¬ 
tion. That is, it should be possible to change the 
physical location of a center from one 
ARPANET host to another. The ability to dy¬ 
namically reconfigure makes it possible to re¬ 
move an improperly operating center from the 
simulator network and replace it with another at 
the same ARPANET host or at a different one. 

4. Decentralization of control: 

McROSS is to be used as a tool for investigating 
distributed computation. Among the subjects to 
be studied are methods for controlling such com¬ 
putations. In particular, various techniques for 
distributing control responsibilities among the 
parts of a computation are to be experimentally 
investigated. It is important, therefore, that 
operation of the McROSS system not require a 
central control mechanism for coordinating 
simulator networks. Stated somewhat differ¬ 
ently, the only components required for a 
McROSS simulation should be simulation cen¬ 
ters defined by McROSS programs. The realiza¬ 
tion of this goal, which makes experimentation 
with distributed control possible, should not 
preclude experimentation with centralized 
control. 

5. Remote monitoring capability: 

A McROSS simulator network should provide 
ports through which its components are acces¬ 
sible to any ARPANET host. An appropriately 
programmed process running at any ARPANET 
host should be able to “attach” to a component 
of a simulator network to monitor and control its 
operation. A remote monitoring process should 
be able to: 

a. obtain sufficient information to display traffic 
in the airspace it is monitoring. 

b. serve as the on-line keyboard for the center 
it is monitoring; 
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c. "detach” from one center and "attach” to 
another in the course of a simulation run. 

McROSS as seen by the user 

A McROSS simulator network is defined by a pro¬ 
gram composed of a "network geometry” sub-program 
and sub-programs corresponding to each of the centers 
in the network. 

The network geometry sub-program defines the logi¬ 
cal geometry for a simulator network. Conceptually, a 
network is composed of nodes and arcs. Nodes in a 
McROSS network are simulation centers and arcs are 
duplex connections between centers. Figure 1 shows a 
four node simulator network which could be used to 
simulate air traffic between Boston and New York. The 
following geometry sub-program defines that network: 

vetb:gin 

nstcen BOSTRM, BOSCEN, NYCEN, 
NYTRM 

netcon BOSTRM, BOSCEN 
nelcon BOSCEN, NYCEN 
netcon NYCEN, NYTRM 
netend 

The netcen statement declares that the network con¬ 


tains four nodes (Boston terminal control, Boston en 
route control, New York en route control and Now 
York terminal control). The netcon statements declare 
the three arcs; nclbcgin and netend serve to bracket the 
geometry declarations. 

In general, the sub-program for each center has four 
parts: 

• a route procedure module 

• a local geography module 

• a wind profile module 

• an aircraft characteristics module. 

In addition to defining procedures followed by air¬ 
craft as they fly through a center’s airspace, the route 
procedure module includes routines specifying how the 
center interacts with its neighbors. Information ex¬ 
change between adjacent centers is accomplished by 
sending messages across the connection between them. 
A center handles messages from neighboring centers by 
submitting them to its input parsing mechanism. Such 
messages arc treated identically to input from its on-line 
console and scenario file. 

The ability of adjacent centers to interact depends 
upon the state of the connection between them as each 
sees it from its end of the connection. A center may 
consider a connection to be in one of three states: 



Figure 1—A simulator network which could be used to simulate 
air traffic between Boston and New York 


1. uninitialized: 

the "physical location” of the neighbor is 
unknown; 

2. closed: 

the “physical location” of the neighbor is known 
but the connection is not capable of carrying 
messages (either because it has not yet been 
established or because it has been broken); 

3. open: 

the connection may be used for information ex¬ 
change with the neighbor. 

In the current implementation of McROSS the 
“physical location" of a neighbor includes both the 
ARPANET host the center is running on (e.g., BBN) 
and the identification of the user it is running under 
(e.g., Jones). 

McROSS provides the operations inil, conn, dsconn 
and abort for changing the state of a connection. The 
effect these operations have on the end of a connection 
is illustrated by the state transition diagram of Figure 2. 

Consider the geometry for the Boston-New York 
simulation. Execution of 

conn BOSTRM 
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Figure 2—Transition diagram for the state of the end of a 
connection showing the effect of the operations init, conn, 
dsconn and abort 


within the BOSCEN initiates an attempt to open a 
connector ’ -'♦h BOSTRM. The connection attempt 
succeeds ii a matching conn is executed by the 
BOSTRM center. The effect of executing . 

dsconn NYCEN 

within the BOSCEN center simulator is to break the 
connection between the NYCEN and BOSCEN centers 
by forcing both ends of it into the closed state; abort 
works in an analogous manner. Execution of the init 
operation results in a center-user dialogue in which the 
human user is asked by the center program to specify 
the physical location of the neighbor. 

Two language primitives arc provided for sending 
messages from one center to another. One takes a single 
operand, a message, which it sends to every neighbor 
whose connection is in the open state. The other takes 
two operands, a message and the najrie of a neighboring 
center. If the connection to the center is open, the mes¬ 
sage is sent to the center; otherwise the operation has 
no effect. Because they are submitted to the input 
parsing mechanism care must be taken that messages 
sent to a neighbor are formatted correctly. 

McROSS includes operations which can be used by 
a center to obtain information about the simulator 
network and its immediate neighbors. For example, 
there is a primitive which produces a list of all nodes in 
the network (i.e., all centers declared by the nctccn 
declaration); another one produces a list of all neigh¬ 
boring centers for which connections are in the open 
state. In addition, a center can examine the state of the 
connection to each of its neighbors individually. 

The local geography module defines the airspace of a 
center by specifying names and locations (x-y coordi¬ 
nates) of important geographic features such as naviga¬ 


tional aids, obstructions and airports. In addition it 
includes a declarative statement which names the simu¬ 
lation center. For example, the geography module for 
the BOSTRM center would include the declaration 

aicccn BOSTRM. 

This declaration has the effect of binding the identifier 
THISIS to the name BOSTRM. Thus in the BOSTRM 
center THISIS is bound to BOSTRM while in the 
NYTRM center it is bound to NYTRM. Route pro¬ 
cedures which access the simulator center name can be 
written in terms of THISIS to enable their use at any 
center in a simulator network. 

A properly authorized process at any ARPANET 
host can attach to a center in a McROSS simulator 
network and request output from it and direct input 
to it thereby monitoring and controlling its operation. 
McROSS center programs are prepared to offer two 
kinds of service to remote monitors: 

1. broadcast service: 

Centers continuously broadcast certain informa¬ 
tion to monitors attached to them. Presently 
centers broadcast flight parameters of each air¬ 
craft in their air space (speed, heading, altitude, 
z-position, y-position, acceleration, aircraft id) 
and the (simulated) time. A remote monitor 
can use broadcast information to drive a display 
of traffic in a center’s airspace or it can record 
it for later analysis. 

2. demand service: 

Each center is prepared to respond to certain 
specific requests from monitors. In the current 
implementation a monitor can request that a 
center: 

a. transmit its map of the airspace (which can 
be used as background for displaying the 
center’s air traffic); 

b. stop the continuous broadcast; 

c. resume the continuous broadcast; 

d. treat the monitor as its on-line keyboard by 
directing keyboard output to the monitor 
and accepting input from it; 

e. cease treating the monitor as its on-line 
keyboard; 

f. break its connection with the monitor. 

The monitoring facility has proven useful both for 
debugging and for demonstration purposes. One diffi¬ 
culty a user faces in debugging a multi-center simula¬ 
tion is determining what is happening at centers sus¬ 
pected to be malfunctioning. A monitor, constructed 
appropriately, can serve as a “graphical probe” and be 
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used to watch the operation of first, one suspect center 
and then another. For example, we have used such a 
monitor to follow the trajectory of an aircraft as it 
passes through several centers. 

By enabling processes at arbitrary ARPANET sites 
to observe and control McROSS simulations, the 
monitoring facility provides a mechanism for using 
hardware and software features which arc unique to 
various network installations. By using monitors which 
play an active role in his simulations a McROSS user 
can experiment with different ways of partitioning 
computational and control responsibilities in air traffic 
situations. He could, for example, experiment with a 
monitor built to provide a weather advisory service for 
simulator centers. Such a monitor would presumably 
have access to an on-line weather data base. (A weather 
data base to be accessible through the ARPANET is 
currently being designed. 10 ) To perform its service for a 
center the monitor would attach to the center, request- 
:... T ...e center broadcast aircraft flight parameters 
to it and accept input lines from it. It would then 
“watch” the airspace of the center and send instructions 
to it, as necessary, to vector aircraft around severe 
weather. 

Unless he chooses to do so, the simulation program¬ 
mer need not concern himself with remote monitoring 
beyond specifying at simulation run time which centers 
in his network are to be receptive to remote monitors. 
Monitors themselves are not part of the McROSS 
system. McROSS merely provides a mechanism for re¬ 
mote monitoring. No effort has been made to provide 
linguistic features within the McROSS system to make 
it easy to write programs to do monitoring. 

THE McROSS IMPLEMENTATION 

Some interesting aspects of the McROSS imple¬ 
mentation are discussed in this section. The section 
focuses on strategy rather than detail. The result is a 
simplified but nonetheless accurate sketch of the 
implementation. 


The ROSS implementation 

Implementation of McROSS was accomplished by 
extending the existing ROSS simulation system. A 
ROSS simulation consists of initialization, simulation 
and termination phases. The simulation phase is imple¬ 
mented as a loop. Each pass through the loop repre¬ 
sents a “tick” of the clock which maintains simulation 
time. On each tick the simulator: 

1. parses and interprets input directed to it from 


SOCKETS 



Figure 3—Schematic of center-center connection between ad¬ 
jacent centers A and B. Sxa, R as, Ssx and Rsx arc assigned at 
program translation time; Hx, Hs, Px and Pa are determined 
at run time 


its scenario file and on-line console since the 
last tick; 

2. interprets the route procedure each aircraft is 
currently following; 

3. advances each aircraft along its trajectory tak¬ 
ing into account the aircraft's speed, acceleration 
and heading and the wind profile of the airspace; 

4. directs output generated since the last tick to 
the appropriate devices; 

5. performs actions necessary to maintain the local 
display of its airspace: 

6. increments the simulated time. 

A ROSS simulation can be run either in a "real time” 
mode in which simulated time is locked to real time, 
or in a “hyper fast” mode in which the simulation pro¬ 
ceeds as fast as it can. 

Process/processor binding and center-center connections 

Connections between pairs of simulator network 
centers arc duplex. Each center-center connection is 
implemented by two ARPANET connections. More 
specifically, an open connection between centers A and 
B is realized by the two ARPANET connections (see 
Figure 3): 

Ha • Pa • Sab—>H n. Pn -Rua 
Ha • Pa ■ Rau 4- H ii . Pu • Sba 

To establish such a center-center connection two pairs 
of matching RFCs must be issued by centers A and B. 
To issue the correct RFCs A must know Hu, P n> Rba 
and Sua; similarly, B must know Ha, Pa, Rab and Sab. 
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The host (H) and process (P) components of the 
socket names for a center-center connection cannot be 
determined until run-time because process/processor 
binding is deferred until then. However, the process- 
local components (It and S) of the socket names can be 
prc-assigned and, in fact, the effect of the declarations 
in the network geometry sub-program for a particular 
McROSS network is to do exactly that. 

The process local components for the four socket 
names corresponding to a center-center connection arc 
always the same whereas the host and process compo¬ 
nents may change from run to run or even within the 
same run if either neighbor is involved in a dynamic 
reconfiguration. 

When a center’s end of a center-center connection is 
in the uninitialized state the host and process compo¬ 
nents of the socket names corresponding to the remote 
end are unknown to it. To move its end of a connection 
from the uninitialized state the center engages in a 
dialogue with the user requesting from him the physical 
location of the neighbor. After successfully completing 
the dialogue the center has sufficient information to 
issue the two RFCs required of it to establish the 
connection. 

Center-monitor connections 

The connection between a center C and an attached 
remote monitor M is realized by two ARPANET con¬ 
nections. One of them 

Hc-Pc-Scm—>H m . Pm • Rmc 

is a "broadcast” connection used for continuously 
broadcasting information to M. The other 

[Hc-Pc.R <—]l 

is a "request” connection maintainedtty C in a listening 
state for M to use to make requests of C. Each monitor 
attached to C has its own broadcast connection but 
may share a request connection with other monitors. 

To make a request of C, M connects to the request 
socket, transmits its request over it and then closes the 
request connection, freeing it for use by other monitors. 
The action taken by C of course depends upon the re¬ 
quest. If the request were for the display map, C would 
transmit the map data over the broadcast connection 
to M. 

Before the RFCs required to establish a center- 
monitor connection between C and M can be issued 
C must know Hu, Pm and Rmc and M must know 
He, Pc, Sc m and R. To obtain the required information, 
M and C engage in a connection protocol which is 
similar to the "initial connection protocol” developed 
by the ARPANET network working group. 8 


Each center willing to service monitors maintains a-s 
an "attach” Hocket a send socket in a listening state 
(see figure 4.a). The attach socket, for C could be 
denoted 

[He • Pc • A—*Jl 

The process local component (A) of the name for at¬ 
tach sockets is the same for all centers and is "well- 
advertised.” Therefore, if M knows the physical loca¬ 
tion of C it can issue an RFC for C’s attach socket. 
The effect of such an RFC is to establish the connection 
(Figure 4.b) 

Hc-Pc-A— >Hm-Pm-Rm 

Upon detecting an RFC for its attach socket, C notes 
Hm and Pm and transmits Scm and R over the connec¬ 
tion. Next both C and M break the connection and 


Hr-Pr'A 


II*!enlng 


(o) 


CENTER 

C 



Figure 4—Schematic of connection protocol exchange between 
center C and monitor M 
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issue the RFCs necessary to establish the broadcast 
connection (Figure 4.c) 

He.P cScm^Hm .Pm ■ Rmc 

where RMC = f(RM), a pre-agrecd upon function of Rm. 
C, if it has not done so previously for another monitor, 
sets up the listening connection 

[Hc-Pc-R 4 —]l 

and finally, reestablishes its attach connection so that 
other monitors can attach to it (Figure 4.d). Currently 
the process-local components for ARPANET socket 
names are numbers and f is taken to be 

f(x)=x+2. 

If R m rather than f(R M ) were used for Rmc a race 
condition would result when M and C attempt to estab¬ 
lish the broadcast connection. The race involves the 
rrd'" ’ '"Inch the connection with socket Hm-Pm-Rm 
wouia ou closed and reopened by C and M. In particu¬ 
lar, the current TENEX NCP is such that an attempt 
by C to open the network file corresponding to its end 
of the (broadcast) connection 

He• Pc• Scm—>Hm . Pm • Rm 

before M closes the network file corresponding to its 
end of the connection 

He. Pc • A—>H m • P m • R m 

would fail. Use of f(RM) for Rmc avoids the race. A 
similar race condition, discovered in an early version 
of the ARPANET initial connection protocol, 7 is 
avoided in the current protocol by the same technique. 8 

Process structure at McROSS centers 

A McROSS center is realized by a collection of co¬ 
operating, asynchronously evolving, sequential pro¬ 
cesses. The collection corresponds to a partitioning of 
the center’s responsibilities into more or less indepen¬ 
dent sub-tasks. It includes: 

1. a process (SIM) to perform the ROSS functions; 

2. a process (CONN) for each center-center con¬ 
nection to establish and maintain the connection; 
and 

3. a monitor server process (MONSER) to service 
remote monitors. 

The CONN process at center A corresponding to the 
center-center connection to center B is responsible for 
establishing ARPANET send and receive connections 
with B. After it establishes the center-center connection 



Figure •>—The process hierarchy which implements a McItOSS 
simulation center with n neighbors 


with B the CONN process maintains the connection. 
When messages from B arrive it passes them on to the 
SIM process for parsing. 

The job of the MONSER process at a center is 
twofold: to engage in an initial connection protocol ex¬ 
change with monitors attempting to attach to the 
center and to respond to requests made by attached 
monitors. 

The processes at a center exist in a hierarchy (see 
Figure 5). The hierarchical structure is less the result 
of any one process being more important than any 
other than it is a consequence of the TENEX con¬ 
straint that process groups be hierarchically arranged. 
During initialization the SIM process creates the 
MONSER and CONN processes. Thereafter, the pro¬ 
cesses evolve more or less independently. 

The process structure at each center helps achieve 
autonomy of parts. The CONN processes and the 
MONSER process serve to protect the SIM process by 
insulating it from direct interaction with other centers 
and remote monitors. 

Protocols 

The current implementation of the TENEX NCP 
is such that if a center were to unilaterally break a con¬ 
nection with a neighbor (by closing the two correspond¬ 
ing ARPANET connections) it could leave processes 
in the neighboring center in an irrecoverable state. For 
example, a process in the neighbor sending information 
across the connection at the time it is broken would 
“feel” the close as if it had executed an illegal instruc¬ 
tion. To prevent such situations McROSS centers en- 
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gage in a protocol exchange prior to breaking 
connections. 

The ccnt.cr-crnter protocol is relatively simple. To 
perform an abort or dsconn operation a center sends its 
neighbor a “request for abort” or “request, for dis¬ 
connect” message and waits until it receives an ac¬ 
knowledgment before actually breaking the connection. 
Existence of the center-center protocol has two major 
implications. The first is that a center-center connection 
has more states than the three noted earlier. The addi¬ 
tional states arc transient ones which the connection 
passes through as the center and its neighbor advance 
through protocol exchanges initiated when one attempts 
to change the state of the connection. The transient 
states are invisible to the McROSS user. Immediately 
after a dsconn (or abort ) is initiated the SIM process 
treats subsequent operations involving the connection 
as if the connection were already in the closed (or 
unir/U . „ed) state. The second implication is that 
center-center connections carry “control” messages 
used in center-center protocol exchange in addition to 
“ordinary” messages intended for the receiver’s parsing 
mechanism. The CONN process for each connection 
must be prepared to recognize and respond appropri¬ 
ately to both kinds of messages. 

McROSS centers expect remote monitors to observe 
a center-monitor protocol. In addition to the connec¬ 
tion and request procedures described earlier, the 
center-monitor protocol includes a disconnection pro¬ 
cedure much like the one used in the center-center 
protocol. 

Interprocess communication 

To perform their tasks the processes at a simulation 
center must interact occasionally. For example, the 
arrival of a message from a neighbor requires inter¬ 
action between the SIM and CONN processes. The 
CONN process receives the message and passes it onto 
the SIM process for parsing and interpretation. 

One way the processes interact is through shared 
memory. For example, the SIM and CONN processes 
have read and write access to a shared “connection 
table.” There is an entry in the table for each center- 
center connection which includes the state of the con¬ 
nection, a semaphore* and other information relcvcnt 
to the connection. Use of the table is coordinated by 
strict adherence to a convention which requires every 
“critical” access (every write access and certain read 
accesses) to an entry to be bracketed by P (lock) and 
V (unlock) operations on the entry’s semaphore. 

The situation arising at a center when a neighbor 
attempts to break connection with the center is a 


typical one which requires interprocess communication. 
The CONN process corresponding to the center-center 
connection receives a “request for disconnect” message 
from the neighbor. Center-center protocol requires the 
CONN process acknowledge the request so that the 
neighbor can break the connection. The purpose of the 
protocol exchange is to warn the center that the con¬ 
nection is about to be broken and that it should no 
longer attempt to use it. Therefore before it acknowl¬ 
edges the neighbor’s request it is important that the 
CONN process communicate this information to the 
other processes at its center. The CONN process docs 
this by the following sequence of actions: 

P(CONNECTION SEMAPHORE); 
set connection-state to “not open”; 

V(CONNECTION SEMAPHORE); 
acknowledge neighbor’s request 

As long as processes at the center perform the following 
sequence of actions to send over center-center connec¬ 
tions there is no danger of sending after the connection 
has been closed: 

P(CONNECTION SEMAPHORE); 
if connection-state = open 
then send 

else abort the send; 

V(CONNECTION SEMAPHORE) 

Stated somewhat differently, sending over a center- 
center connection should be regarded as an operation 
which involves a “critical” read access of the corre¬ 
sponding connection table entry. 

In addition to memory sharing, direct process control 
is used as a mechanism for interprocess communication 
in the McROSS system. Because of its superior position 
in the process hierarchy the SIM process can exert 
direct control over the other processes. A few situations 
occur in which it does so. For example, when a center- 
center connection has been closed or aborted (via 
dsconn or abort ) the. SIM process forces the correspond¬ 
ing CONN process to halt. If and when an attempt is 
initiated to reestablish the connection (via conn ) 
SIM restarts it. 

SOME OPEN QUESTIONS 

This section briefly discusses questions representa¬ 
tive of ones which have arisen in the course of using 
the McROSS system. The questions have been resolved 
to the extent that useful simulations can be performed 
using McROSS. However, none has been resolved in a 




McROSS 291 


totally satisfactory manner. The intent of this section 
is to leave the reader with an appreciation for the issues 
raised by these questions; a thorough discussion of 
them is well beyond the scope of this paper. 

Synchronization 

Simulated time is important in the operation of the 
McROSS system. In particular, whenever an inter¬ 
action between adjacent centers occurs it is important 
that the clocks kept by the centers show approximately 
the same time. Time synchronization is a specific ex¬ 
ample of the general problem of control in distributed 
computation. It is compounded by the fact that centers 
can start up and shut down independently of one 
another. A centralized approach to synchronization 
has been used with success in McROSS simulations. 
In it, one Center acts as a synchronizer for an entire 
simulator network. When a center starts up it connects 
to the synchronizer and receives a synchronization 
message from it. Thereafter, to stay in synch with 
ether centers in the network, the center makes use of 
the real time clock in the computer it runs on. A dis¬ 
tributed approach to synchronization which docs not 
require a synchronizing center is under consideration. 

Locally unknown names 

Names that are well defined within a simulator net¬ 
work as a whole are not necessarily defined at every 
node in the network. How should references to such 
names occurring within centers in which they are not 
defined be handled? For a specific example in which 
such a reference is reasonable, reconsider the four node 
network for simulating Boston-New York traffic. ■ A 
user controlling the simulator for Boston Terminal who 
is manually vectoring an aircraft leaving Logan airport 
might reasonably issue the clearance 

fly (V205, PAWLING) 

which specifies that the aircraft is to follow Victor Air¬ 
way #205 to Pawling. Assume that V205 is defined 
within the geography modules for BOSTRM, BOSCEN 
and NYCEN and the PAWLING is defined within 
NYCEN but not within BOSTRM or BOSCEN. The 
BOSTRM center can’t fly the aircraft to Pawling be¬ 
cause Pawling is not defined within its airspace. Ideally 
it should fly the aircraft along Y205 to the boundary 
of the BOSCEN airspace and then hand it. off to the 
BOSCEN simulator. Certainly it should be able to do 
better than report an error and abort the route pro¬ 
cedure. Techniques for handling references to locally 


unknown names in certain limited contexts are being 
investigated. However, the general problem of handling 
such references is an open question. 

Program residence 

Where should the program (route procedures) re¬ 
quired to fly an aircraft through several simulator 
centers reside? Should the program be associated with 
the aircraft and passed with it from center to center or 
should parts of the program be distributed among the 
relevant centers? The approach currently used in 
McROSS simulations is to distribute the program and 
pass only the aircraft and its flight parameters from 
center to center. 

Interruption and resumption of route procedures 

Aircraft frequently interrupt their flight plans tem¬ 
porarily in order to avoid severe weather or heavy 
traffic. The simulation analogy to a flight plan is the 
route procedure. How should a center handle interrup¬ 
tion and subsequent resumption of route procedures? 
Interrupting the execution of a route procedure in 
order to follow another one is not difficult. The diffi¬ 
culty arises in determining how to appropriately resume 
the interrupted procedure. In general, the point of in¬ 
terruption is inappropriate as a resumption point. A 
considerable amount of (simulated) time can elapse 
between interruption and resumption during which the 
flight parameters (position, speed, altitude and head¬ 
ing) of the aircraft can change significantly. Therefore, 
the usual programming technique of saving the “state” 
when an interrupt occurs and restoring it after the 
interrupt has been handled is inadequate. The inter¬ 
ruption/resumption problem is made more complex by 
the possibility that between interruption and resump¬ 
tion the aircraft may fly out of the airspace of one 
center and into the airspace of another. The current 
McROSS implementation is not prepared to handle 
interruption and subsequent resumption of route 
procedures. 

Error handling techniques for distributed systems 

The question of how to handle error situations in a 
distributed computational system is a challenging one. 
In McROSS considerable attention has been given to 
making nodes in a simulator network autonomous. The 
strategy for handling errors is to try to achieve a 
“local” error recovery whereby a node attempts to pre¬ 
serve its autonomy. As a result, while the actions it 
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takes are locally optima! in tin- sense that its continued 
operation is insured, they may he sub-optimal in the 
more global context of the entire- simulation network. 

Errors occurring in inter-node messages are simply 
handled in the current McItOSS implementation. Re¬ 
call from an earlier section that inter-node messages 
are submitted to the parsing mechanism of the destina¬ 
tion node. When a node receives a message which is 
syntactically incorrect or semantically meaningless 
(to it) from a neighbor, it reports the error on its 
on-line keyboard, sends a message to the neighbor caus¬ 
ing the error to be reported on the neighbor’s on-line 
keyboard, and ignores the message. This procedure is 
locally satisfactory in the sense that it guarantees that 
messages which are not well-formed cannot cause the 
node to malfunction. However, if the incorrect message 
from the neighbor is one critical to the outcome of the 
simulation, this procedure is not globally acceptable. 
Ideally, upon detecting an error in a message, the node 
should engage in a dialogue with its neighbor in an 
attempt to resolve the error. The difficulty in imple¬ 
menting this strategy is that it is frequently unclear 
what should be done to resolve an error. Often the 
cause has been masked by the time the error is detected. 

While the simple techniques used in McROSS for 
error handling have proven adequate, it is clear that 
more effective techniques can be developed. 

CONCLUDING REMARKS 

The results of the work reported in this paper are ap¬ 
plicable in two areas. 

One area is research in air traffic control. Researchers 
can use the McROSS system to conduct simulation 
studies preliminary to the design of an automated 
multi-component air traffic control, system. For ex¬ 
ample, McROSS could be used to evaluate various 
ways of partitioning responsibility among components 
of such a system. Or, it could be used to compare differ¬ 
ent strategies for automated scheduling and control of 
aircraft. Because it exhibits autonomy of parts and the 
ability to dynamically reconfigure, it could be used to 
experimentally study the behavior of failure handling 
techniques for a multi-center system under various air 
traffic loads. 

The other area is the design and implementation of 
distributed computation systems. The results in this 
area are incomplete and tentative consisting primarily 
of insights and attitudes developed from the experience 
of building and using a distributed computation system. 
These are summarized by reviewing the goals fur the 
McROSS system in terms of problems they posed and 
techniques useful in realizing them. 

Of the five goals, autonomy of parts and deferral of 


process/processor binding were the most significant in 
terms of effort required to achieve them and influence 
on the appearance of the system to users. Given their 
realization, the other three goals (the ability to dy¬ 
namically reconfigure a simulator network, decentrali¬ 
zation of control and ports for monitoring) were rela¬ 
tively easy to achieve. 

The strategy of implementing parts of the distributed 
computation by process groups rather than solitary 
processes contributed significantly to achieving au¬ 
tonomy of parts. The multi-process implementation 
made it possible to dedicate certain processes at a node 
to maintaining an interface with other nodes and to 
dedicate other processes to performing functions crucial 
to the node’s existence. In addition to insulating vital 
internal node functions from the actions of other nodes, 
the functional modularity resulting from multi-process 
nodes had the effect of reducing the complexity of the 
implementation: each individual process being con¬ 
siderably less complex than an entire node. The multi¬ 
process capability which the TENEX operating system 
supports for each user job was invaluable in carrying 
out the multi-process strategy. It is unfortunate that 
few- operating systems allow users to create and main¬ 
tain process structures. 

Equally useful in realizing autonomy was the estab¬ 
lishment of and strict adherence to protocols for part- 
part interactions. A center can expect monitors and 
adjacent centers which arc functioning properly to ob¬ 
serve protocol and can therefore interpret a breach of 
protocol as a warning that the offender may be mal¬ 
functioning. A consequence of the multi-process imple¬ 
mentation of nodes is that interprocess communication 
occurs within McROSS at two levels: inter-node and 
intra-node. Use of a protocol for intra-node interactions 
helps insure that internal node data bases always ac¬ 
curately reflect the condition of the node’s interface 
with other nodes. A useful implementation rule was to 
reject any technique whose success depended upon the 
order in which events in different centers or in different 
processes within the same center occur. 

The major problem in implementing deferred process/ 
processor binding was providing a way for parts of the 
computation to determine the location of logically ad¬ 
jacent parts at run time. The solution used in the 
current McROSS implementation, which requires run 
time interaction with the user, is not totally satisfac¬ 
tory. A more satisfactory alternative might be for each 
part to engage in a network-wide search for logically 
adjacent parts. 

We expect to see a trend toward distributed multi¬ 
computer systems in the future. By its existence 
McROSS demonstrates that the construction of such 
systems is currently feasible. 
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Description of McROSS Simulator^Network: 



netbegln 

netcen NYTRM, BOSTRM, NYCEN 
BOSCEN, CLEVCEN; 
n>tcon ( NYTRM, NYCEN ) ; 
netcon ( NYCEN, BOSCEN ) ; 
netcon ( CLEVCEN, NYCEN ) ; 
netcon (CLEVCEN, BOSCEN ) ; 
netcon ( BOSTRM, BOSCEN) ; 


netend 






User Defined Airspace for Boston Terminal Area 



Components of McROSS System 

•ROSS 

• TENEX Operating System 


•ARPANET 



BOSCEN and BOSTRM Establish Connection by: 


( at BOSCEN ) 


(at BOSTRM) 


ln_lt ( BOSTRM, BBN, SMITH); J_n_[t ( BOSCEN, UTAH-10, JONES); 


conn (BOSTRM) ; 


conn ( BOSCEN); 


Configuration During a Typical McROSS Simulation Run 
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Extensions of packet communication technology 
to a hand held personal terminal 


by LAWRENCE G. ROBERTS 

A dvanccd Research Project* A gency 
Arlington, Virginia 


INTRODUCTION 

Electronic communications technology has developed 
historically almost completely within what might be 
called the circuit switching domain. Not until the last 
decade has the other basic mode of communication, 
packet switching, become competitive. Thus, as a tech¬ 
nology, packet communication has only begun to be 
explored. Circuit switching can be defined in the broad 
sense as the technique of establishing a complete path 
between two parties for as long as they wish to com¬ 
municate, whereas packet switching is where the com¬ 
munication is broken up into small messages or packets, 
attaching to each packet of information its source and 
destination and sending each of these packets off inde¬ 
pendently and asynchronously to find its way to the 
destination. In circuit switching all conflicts and alloca¬ 
tions of resources must be made before the circuit can 
be established thereby permitting the traffic to flow 
with no conflicts. In packet switching there is no dedi¬ 
cation of resources and conflict resolution occurs during 
the actual flow perhaps resulting in somewhat uneven 
delays being encountered by the traffic. Clearly, without 
the speed and capability of modern computers, circuit 
switching represented a cheaper and more effective 
way to handle communications. For radio frequency 
assignment and telephone exchanges the resource allo¬ 
cation decisions could be made infrequently enough 
that manual techniques were originally sufficient. Also, 
since voice was the main information being communi¬ 
cated, the traffic statistics were sufficiently compatible 
with this approach to make it quite economic for the 
period. Packet switching of a kind, the telegram, per¬ 
sisted throughout this period but due to the high cost 
of switching and the limited demand for fast message 
traffic never attracted much attention. 

For almost a century circuit switching dominated 
the communications field and thus dominated the de¬ 
velopment of communications theory and technology. 


Now, within the last decade or less, the advances in 
digital computers and electronics have, in many eases, 
reversed the economic balance between circuit and 
packet communication technology. Perhaps the best 
proof of this is the economy of the ARPA Network 1- ' 
for country-wide computer to computer communica¬ 
tion, but many other examples are beginning to appear 
such as the University of Hawaii’s ALOHA System 7 
utilizing packet radio transmission for console com¬ 
munications and the experiments with digital loops for 
local distribution. However, most of the experiments 
with packet communications have been undertaken by 
computer scientists, and it is not even generally recog¬ 
nized yet in the communications field that a revolution 
is taking place. Even where the knowledge of one of 
these experiments has penetrated the communications 
field, it is generally written off as a possibly useful new 
twist in communications utilization, and not recognized 
as a very different technology requiring a whole new 
body of theory. Throughout the development of the 
ARPA Network, communication engineers compared it 
with conventional circuit switched systems but, per¬ 
haps unconsciously, used rules of thumb, statistics and 
experience applicable only to circuit switched systems 
as a basis for comparison. A century of experience and 
tradition is not easy to ignore and in fact should not be 
ignored, only it should be properly classified and segre¬ 
gated as resulting from a different technology. 

Packet communication technology is only beginning 
to be explored but already it is clear that the design of 
all forms of communications channels and systems 
should be rethought. As an example of the kind of 
difference packet communications can make in a per¬ 
haps unexpected area, the design of a personal terminal 
will be explored in some detail. Although such a ter¬ 
minal has never been built, it is most likely completely 
feasible to build and would provide many unique 
advantages. 
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HAND HELD PERSONAL TERMINAL 

Let us start with the gold of providing each individual 
with a pocket-sized, highly reliable and secure com¬ 
munications device which would permit him t.o send 
and receive messages to other individuals or to co- 
puters. Leaving the consideration of design alternatives 
muntil the end, a device fulfilling these objectives is as 
follows; 

Output 

Text or graphics displayed on a 2.S"Xl" plasma 
panel with SO dots per inch resolution. The screen, 
divided into 7X10 dot rectangles, using 5X7 characters 
would hold 8 lines of 32 characters each for a total of 
25G characters. Text this size is almost the same size as 
typewriter print, except that the lines are closer to¬ 
gether. The plasma panel has internal storage and is 
digitally addressed to write or erase a point. 

Input 

Five capacity or stress sensitive buttons used by the 
five fingers of one hand simultaneously to indicate one 
of 31 characters. This five finger keyboard technique 
was developed by Doug Englebart at SRI 8 to permit 
users to type wdth only one hand while working on a 
display console. Recently the keyboard has become 
fairly widely used at SRI due to its great convenience. 
Training time for a new user is evidently less than a 
day and speeds of 30 words per minute can be achieved. 8 
Although somewhat slower than a good typist (H 
speed) the speed is clearly sufficient for a terminal 
device even at 10 words/minute.' ■= 

Transmission 

Each input character will be transmitted to a central 
controller station using the random access radio trans¬ 
mission techniques developed at the University of 
Hawaii. 7 The 5 bit character is embodied in a 04 bit 
packet containing; 

30 bits—Terminal Identification Number 
8 bits—Character plus alternation bit, or Count 
2 bits—Type of packet (CHAR, ACK, CNT, 
ERR, ST ERR) 

24 bits—Cyclic Sum Check 
64 bits 

All terminals transmit their packets independently and 
asynchronously on a single frequency and the receiver 


at the central controller merely listens for a complete 
packet which has a correct, sum check. If two terminals' 
transmissions overlap the sum check will be wrong, and 
the terminals will retransmit when they find they don’t 
receive an acknowledgment. Retransmission time-out 
intervals are randomized between the terminals to 
avoid recurrence of the problem. Upon receipt of a good 
packet, the central station transmits a display-ac¬ 
knowledgment packet back to the terminal on a second 
frequency. This 144 bit packet contains a 70 bit display 
raster field and an 8 bit position on the screen. The dis¬ 
play raster is a 7 X 10 dot array for the character sent 
in and the position includes 3 bits for vertical by f> bits 
for horizontal. Current position information for each 
active user is kept by the central station by user ID in 
a hash table. Thus, the individual terminal needs no 
character generation logic, position advancement logic, 
or any other local verification of the input since the 
response from the central station both acknowledges the 
character and displays it in an input text line at the top 
of the display. If a character display-acknowledgment 
is somehow lost in transmission the terminal will con¬ 
tinue to time-out and retransmit the character. The 
central station must somehow differentiate this from 
a new character. This is achieved by an alternation 
bit 10 -’ 1 in the terminal’s packet which is complemented 
for each new character. On a repeat the bit is the same 
as previously and the central station just retransmits 
the same character and position again. When a pre¬ 
arranged terminating character is sent the central 
station examines the message and takes an appropriate 
action. Considerable flexibility exists here, and opera¬ 
tional modes could be established. However, the first 
message of a sequence should contain a destination as 
the first item. This might be the. ID of another terminal 
in the same area, it might be the address of a service 
computer or it might be the ID of another terminal 
halfway around the world. In the latter two cases, a 
more global network such as the ARPA Network comes 
into play. It would be perfectly feasible for a message 
to another terminal to be sent to a central or area-coded 
directory computer to locate the particular control 
station the other terminal was near. Note that the loca¬ 
tion of neither man was given to the other, only the 
message and the ID of the sender. (Based on ARPA 
Network cost estimates and international satellite tariff 
trends, such a message exchange should cost less than 
0.1 cents, independent of distance.) 

Reception 

At any time when a message destined for a terminal 
arrives at the central control station, a transmission to 
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the terminal may begin, character by character, each 
in its own 144 bit packet as follows: 

30 bits—Terminal Identification Number 
70 bits—7X 10 dot pattern, character display 
8 bits—position of character 
1 bit —alternation bit 
1 bit —broadcast mode 

3 bits—Message Type (Response, initial, 
normal) 

8 bits—Characters Left in message 
24 bits—Cyclic Sum Check 
144 bits 

The terminal must always be checking all transmission 
to detect those with its ID and a correct sum check. 
When one occurs which is not a “response” to an input 
character, a message is being sent. The first character 
of a message is marked type “initial,” and has the 
count of the remaining characters. Each character is 
displayed where the central station placed it. Following 
the "initial” character “normal” characters arc checked 
to make sure the count only decreases by one each time. 
After the character with count zero, an acknowledg¬ 
ment type packet is sent by the terminal. If this is lost 
(as it may be due to conflicts) the central control will 
retransmit the final character over again without com¬ 
plementing the alternation bit until it is acknowledged 
(or it determines the station is dead). If a count is 
skipped the terminal sends a CNT ERR message with 
the count of the character expected. The transmitter 
then starts over at that count. If a “normal” type char¬ 
acter is received before an “initial” type a ST ERR 
message is sent and the message is restarted. A broad- 
case bit is included which overrides the ID check for 
general messages. 

Security 

Since all transmissions arc digital, encryption is pos¬ 
sible and would be important no matter what the appli¬ 
cation, military or civilian. Most private uses such as 
personal name files, income-expense records, family 
conversations, etc., would be far more sensitive than 
current computer console use. 

Bandwidth 

Personal terminals for occasional use for message ex¬ 
change, maintaining personal files, querying computer 
data bases for reference data, etc., would not lead to 
very heavy use, probably no more than two query- 
rosponsos per hour. The query we might estimate, at 04 
characters in length and the response at 250. (Clearly 


250 character response could also consist of an 80 X 
224 point graphic display since each character is sent as 
a full 7X10 raster.) The average bandwidth consumed 
by each terminal is therefore 2.3 bits/sccond trans¬ 
mitted and 25.0 bits/second received. The random ac¬ 
cess technique used for transmission requires the chan¬ 
nel bandwidth to be six times the average bandwidth 
actually utilized in order to resolve all conflicts prop¬ 
erly. Thus, the terminal transmission bandwidth con¬ 
sumption is 14 bits/second, still less than the receiver 
bandwidth needed. Thus, the central control station's 
transmitter bandwidth is the limiting factor assuming 
equal bandwidths on both transmitter and receiver. If 
a 50KHz bandwidth is used for each and modulated 
at 50K bits/scc, then a total of 2000 terminals can be 
accommodated. Of course this number depends on the 
activity factor. At one interaction every two minutes a 
data rate equal to average time shared console use is 
obtained and even at this activity 130 terminals can be 
supported, more than most time-sharing systems can 
support. With 50 KB channels, the time required to 
write 256 characters is about one second. Lower band- 
widths require increased time, thus, 10KB (5 sec write 
time) would be the lowest bandwidth reasonable. Even 
at this bandwidth, with the estimated 2 interactions 
per hour, 400 terminals could be supported. 

COMPARISON 

Comparing the effect of the packet technology’ with 
the same terminal operating with preassigned Fre¬ 
quency or Time Division Multiplexed channels (ignor¬ 
ing the losses due to TDM sync bits or FDM guard 
bands) the circuit oriented terminal would require a 
40 bit/sec transmit channel and a 4KB receive channel 
if a 5 sec write time is to be achieved. For 400 terminals 
with a 5 sec write time, the circuit method would re¬ 
quire a total of 1.6 Mcgabits/scc bandwidth whereas the 
packet method only requires 20 Kilobits/sec band¬ 
width. Thus, the circuit technology requires a factor of 
80 more bandwidth than the packet technique. Of 
course, the circuit mode terminals could interact more 
often within the same bandwidth right up to continual 
rewrite of the display every five see, but y’ou would 
have to massively reshape the user statistics to suit the 
technology. 

Another possibility, to design the terminal so that it 
performed more effectively in a circuit oriented mode, 
would be to put character generation logic and position 
logic in the terminal. This would considerably increase 
the cost of the terminal, which originally’ had very little 
logic except shift registers. The result of adding this 
logic, however, is to reduce the bandwidth by a factor 
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of 10 t.o ,10MB or .still 8 times the packet, technique. 
The same logic would help reduce the packet size hut., 
in order to maintain the graphic output capability and 
gross simplicity, it does not seem to pay. 

CONCLUSION 

As can be seen from the. example, packet technology is 
far superior to circuit technology, even on the simplest 
radio transmission level, so long as the ratio of peak 
bandwidth to average, bandwidth is large. Most likely, 
the only feasible way to design a useful and economi¬ 
cally attractive personal terminal is through some typo, 
of packet communication technology. Otherwise one is 
restricted to uselessly small numbers of terminals on 
one channel. This result may also apply to many other 
important developments, only to be discovered as the 
technology of packet communication is further 
developed. 
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INTRODUCTION 

A little over three years ago the Advanced Research 
Projects Agency of the Department of Defense (ARPA) 
began implementation of an entirely new venture in 
computer .communications: a network that would 
allow for the interconnection, via common-carrier 
circuits, of dissimilar computers at widely separated, 
ARPA-sponsored research centers. This network, which 
has come to be known as the ARPA Network, presently 
includes approximately 20 nodes and is steadily growing. 
Major goals of the network are (1) to permit resource 
sharing, whereby persons and programs at one research 
center may access data and interactively use programs 
that exist and run in other computers of the network, 
(2) to develop highly reliable and economic digital 
communications, and (3) to permit broad access to 
unique and powerful facilities which may be eco¬ 
nomically feasible only when widely shared. 

The ARPA Network is a new kind of digital com¬ 
munication system employing wideband leased lines 
and message switching, wherein a path is not es¬ 
tablished in advance and instead each message carries 
an address. Messages normally traverse several nodes 
in going from source to destination, and the network is 
a store-and-forward system wherein, at each node, a 
copy of the message is stored until it is safely received 
at the following node. At each node a small processor 
(an Interface Message Processor, or IMP) acts as a 
nodal switching unit and also interconnects the re¬ 
search computer centers, or Hosts, with the high band¬ 
width leased lines. 

A set of papers presented at the 1970 SJCC' -6 
described early work on the ARPA Network in some 
detail, and acquaintance with this background ma¬ 
terial (especially Reference 2) is important in under- 
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standing the current _work. The present paper first 
discusses major developments that have taken place 
in the network over the last two years. We then de¬ 
scribe the Terminal IMP, or TIP, a development 
which permits direct terminal access to the network. 
Finally we mention some general issues and discuss 
plans for the next stages in development of the network. 

THE DEVELOPING NETWORK 

The initial installation of the ARPA Network, in 
1969, consisted of four nodes in the western part of 
the United States. A geographic map of the present 
ARPA Network is shown in Figure 1. Clearly, the 
most obvious development has been a substantial 
growth, which has transformed the initial limited experi¬ 
ment into a national assemblage of computer resources 
and user communities. The network has engendered 
considerable enthusiasm on the part of the participants, 
and it is increasingly apparent that the network repre¬ 
sents a major new direction in both computer and 
communications technology. 

Figure 2 is a logical map of the network, where the 
Host computer facilities are shown in ovals, all circuits 
are 50 kilobits, and dotted circuits/nodes represent 
planned installations. On this figure certain nodes are 
listed as a "316 IMP”; this machine is logically nearly 
identical to the original IMP, but can handle ap¬ 
proximately two-thirds of the communication traffic 
bandwidth at a cost savings of approximately one-half. 
The original IMP includes a Honeywell 516 computer, 
and more recently Honeywell began to market the 316 
computer as a cheaper, downward-compatible machine. 
As the network has grown, sites were identified which 
did not require the full bandwidth of the original IMP, 
and a decision was made to provide an IMP version 
built around the 316 computer. Also shown in Figure 2 
are certain nodes listed as "TIP”; this new machine 
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Figure 1—ARPA Network, geographic map, December 1971 


is discussed in detail later in this paper. Site abbrevia¬ 
tions shown on Figures 1 and 2 are explained in Table I. 

As the network has grown, a great deal of work has 
been concentrated on the development of Host-to- 
Host protocol procedures. In order for programs 
within one Host computer system to communicate 
with programs in other Hosts, agreed-upon procedures 
and formats must be established throughout the net¬ 
work. This problem has, as predicted, turned out to be 
a difficult one. Nonetheless protocol procedures have 
evolved and are being accepted and implemented 
throughout the net. At the present writing, many of 
the Hosts have working “network control programs’’ 
which implement this protocol. Protocol development 
is more fully reported in a companion paper,' but we 
wish to make a general observation on this subject: 
the growth of the network has dynamically catalyzed 
an area of computer science which has to do with the 
quite general problem of how programs should inter¬ 
communicate, whether in a single computer or between 
computers. Thus the evolution of the Host-to-Host 
protocol represents a side benefit of the network that 
reaches well beyond its utility to the network alone. 



Figure 2—ARPA Network, logical mup, December 1971 


Since both hardware and software network con¬ 
nections must be implemented by each Host, it is 
important that the external characteristics of the 
IMP be relatively stable. This stability has been care¬ 
fully maintained, while at the same time internal 
operation of the IMP program has undergone extensive 
revision and improvement. For example, trouble 
reporting, statistics gathering, and test procedures 
have been substantially improved. In addition to im¬ 
provements that have already been incorporated into 
the program, there have also been extensive studies of 
performance and message flow control. 7 These studies 
have pointed up areas of vulnerability to perverse 


TABLE I—Site Abbreviations 


NCAR 

National Center for Atmospheric Research, 

GWC 

Global Weather Central 

SRI 

Stanford Research Institute 

MC CLELLAN 

McClellan Air Force Base 

UTAH 

University of Utah 

ILLINOIS 

University of Illinois 

MIT 

Massachusetts Institute of Technology 

LINCOLN 

M.I.T. Lincoln Laboratory 

RADC 

Rome Air Development Center 

CASE 

Case Western Reserve 

AMES 

N.A.S.A. Ames Research Center 

use 

University of Southern California 

UCSB 

University of California at Santa Barbara 

STANFORD 

Stanford University 

SDC 

Systems Development Corporation 

BBN 

Bolt Beranek and Newman 

CARNEGIE 

Carnegie University 

MITRE 

MITRE Corporation 

ETAC 

Environmental Technical Applications 
Center 

UCLA 

University of California at Los Angeles 

RAND 

Rand Corporation 

TINKER 

Tinker Air Force Base 

HARVARD 

Harvard University 

BURROUGHS 

Burroughs Corporation 

NBS 

National Bureau of Standards 


heavy traffic patterns and have suggested still other 
possible improvements in the routing and flow control 
mechanisms. Potential changes are presently being 
being studied in some detail and will be incorporated 
into one or more forthcoming major ’revisions of the 
program. They will hopefully anticipate problems 
which might be expected to arise as traffic flow in the 
network becomes heavier. 

Somewhat belatedly in the network design, the need 
to connect a single IMP to several Hosts was recog¬ 
nized. This required multiple Host interfaces at the 
IMP as well as more complex IMP software. Further, 
the various Host computers at a site are often physically 
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distant from one another, thus requiring an increase in 
the maximum allowable physical separation between a 
Host and its IMP. To connect to an arbitrarily remote 
Host would have meant a communications interface 
capable of attachment to common-carrier circuits via 
modems. It would furthermore have required coopera¬ 
tive error control from the Host end. At the time, we 
chose not to modify the logical way in which the IMP 
dealt with the Host and instead we provided more 
sophisticated line drivers which would handle distances 
of up to two thousand feet. Several such "Distant Host” 
installations are now working in the network. Un¬ 
fortunately, as the network has grown, new sites have 
appeared where still greater Host/IMP distances are 
involved. The present scheme does not include error 
control, and use of this scheme over greater distances 
is not appropriate. At the present time we are therefore 
considering how best to arrange IMP/Host connec¬ 
tions over large distances and additional options will 
be required in this domain. 

Another facility which has been tested is the ability 
of the IMPs to communicate over 230.4 kilobit phone 
lines instead of 50 kilobit lines. A short, fast link was 
incorporated into the network for a brief period and 
no problems were encountered. To date, network 
loading has not justified upgrading any operational 
network circuits to 230.4 kilobits, but this will be con¬ 
sidered as loading rises. 

Substantial effort has gone into traffic and trouble 
reporting. A Network Control Center (NCC) has been 
built at Bolt Beranek and Newman Inc. in Cambridge, 
where a small dedicated Host computer receives re¬ 
ports each minute from every IMP on the network. 
Traffic summaries and status and trouble reports are 
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Figure 3—Typical segment of N'CC log 


then generated from this material. Specifically, a 
logger records any changes that the IMPs report to the 
NCC; it records line errors and IMP storage counts 
when they exceed certain limits, as well as unusual 
events, such as reloading from the net, checksum errors, 
etc. Figure 3 shows an example of this log. (The com¬ 
ments, in parentheses to the right, are not part of the 
log but have been added to explain the meaning of the 
entries.) In addition to this detailed log of interesting 
events, one may at any time obtain a quick summary 
of the status of the network. Finally, detailed and 
summary logs of Host and line traffic are produced. 
The NCC is a focal point not only for monitoring the 
network but also for testing and diagnosing remote 
troubles. Lines throughout the network can be looped 
from here in order to isolate difficulties. Personnel of 
the center coordinate all debugging, maintenance, 
repair and modification of equipment. 


DIRECT TERMINAL ACCESS 

During the early phases of network development 
a typical node has consisted of one or more large time- 
shared computer systems connected to an IMP. The 
IMPs at the various sites are connected together into 
a subnet by 50 kilobit phone lines and the large Host 
computers communicate with one another through 
this subnet. This arrangement provides a means for 
sharing resources between such interconnected centers, 
each site potentially acting both as a user and as a 
provider of resources. This total complex of facilities 
constitutes a nationwide resource which could be made 
available to users who have no special facilities of their 
own to contribute to the resource pool. Such a user might 
be at a site either with no Host computer or where the 
existing computer might not be a terminal-oriented 
time-sharing system. 

A great deal of thought went into considering how 
best to provide for direct terminal access to the net¬ 
work. One possibility, which would have essentially 
been a non-solution, was to require a user to dial direct 
to the appropriate Host. Once connected he could, of 
course, take advantage of the fact that that Host was 
tied to other Hosts in the net; however, the network 
lines would not have been used to facilitate his initial 
connection, and such an arrangement limits the ter¬ 
minal bandwidth to what may be available on the 
switched common-carrier networks. 

A similar solution was to allow terminals to access 
the network through a Host at a nearby node. In such 
a case, for example, a worker in the New England area 
wishing to use facilities at a California site might 
connect into a local Boston Host and use that Host 
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as a tap into the network to get at the facilities in 
California. This approach would have required Hosts 
to provide hardware access facilities for many ter¬ 
minals which would use their systems only in passing. 
For many Hosts, the kinds of terminals which can be 
connected directly are limited in speed, character set, 
etc. In terms of reliability, the user would have been 
dependent on the local Host for access: when that Host 
was down, his port into the network would be un¬ 
available. Furthermore, the Hosts would have been 
confronted with all of the problems of composing 
terminal characters into network messages and vice 
versa as well as directing these messages to the proper 
terminals and remote Hosts. Time-sharing systems are 
generally already overburdened with processing of 
characters to and from terminals and many are con¬ 
figured with front end processors employed explicitly 
to off-load this burden from the main processor. In¬ 
creasing the amount of such work for the Hosts there¬ 
fore seemed unreasonable and would have resulted in 
limiting terminal access. Instead, a completely separate 
means for accessing the network directly seemed called 
for: an IMP with a flexible terminal handling capa¬ 
bility—a Terminal IMP, or TIP. 

One of the fundamental questions that arises when 
considering this problem is: what is the proper dis¬ 
tribution of computational power among the remote 
big facility, the terminal processor, and the terminal? 
Shall the terminal processor be clever, have sizable 
storage, be user programmable, etc., or shall it be a 
simpler device whose basic job is multiplexing in a 
flexible way? Serious work vrith interactive graphics 
seems to require the terminal to include, or be in 
propinquity to, a user-programmable processor and 
considerable storage. To date, such work has primarily 
been done with terminals attached directly to a Host. 
Some elaborate terminals, such as the Adage, include 
a powerful processor. Other kinds of terminals, in- 
eluding Teletype-like devices, alphanumeric displays, 
or simple graphic displays (excluding serious inter¬ 
active graphics), do not require a user-programmable 
local processor or significant local storage. 

We believe that the great majority of potential 
groups needing access to the ARPA Network will not 
need powerful interactive graphics facilities. Further, 
for the minority that will need powerful terminals, we 
believe that individual terminals with built-in or 
accompanying processors will become more common. 
For these reasons, we decided that the terminal pro¬ 
cessor should be simple and not programmable from 
the terminals. The computational load and the storage 
should be in the IIosts or in the terminals and not in the 
terminal processor. This simple multiplexing approach 
is amenable to some standardization and is pliiio- 


sophically close to the original IMP notion of a stan¬ 
dard nodal device. 

Another major question we faced was whether to 
build a separate terminal handler and then connect it 
to the IMP or to build an integrated unit that was 
housed in a common cabinet and used the IMP pro¬ 
cessor. One advantage of the two-machine approach 
is that it isolates the IMP functions from the terminal 
functions, thereby providing a barrier of safety for the 
net. This approach also provides the processing power 
of two machines and a potentially greater degree of 
user freedom in modifying or writing programs. 
Another interesting reason for considering separate 
machines was to reduce the large cost associated with 
I/O equipment (such as line controllers) by making 
use of the extra processing power. We discussed with 
several manufacturers the possibility of bringing 
terminal wires “into” their processors and decoding 
the basic line information directly in software. How¬ 
ever, even with some of the new state-of-the-art 
machines, like the Meta-4, with fast 90 nsec read-only 
memories to handle character decoding, the I/O cost 
was still high, and in large part the necessary I/O 
equipment was yet to be designed. We therefore con¬ 
cluded that it was still somewhat early to proceed in 
this fashion, and two processors did not appear to save 
I/O equipment. 

The principal disadvantages of the two-machine 
approach were the higher initial cost, the difficulties 
of maintaining two machines, and the software problems 
of dealing with two machines. In particular, the com¬ 
munication between two machines would require two 
hardware Host interfaces, and two software Host/IMP 
programs. This would result in a much poorer com¬ 
munication between the IMP program and the ter¬ 
minal handling program than would exist in a single 
machine. In either case, one machine or two, the 
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Figure 4—A TIP in the network 



The Terminal IMP 247 



terminal handling process required the implementa¬ 
tion of a version of Host protocol. 

We finally decided that the least expensive and most 
sensible technical alternative was to build the IMP 
and the terminal handler (with the necessary subset 
of Host protocol) together into a single machine. Then, 
since certain new machines -appeared attractive in 
cost/performance relative to the Honeywell 16 series, 
we spent considerable time thinking about what 
machine to use. We decided that no alternate choice 
was sufficiently attractive to justify rewriting the 
main IMP program and redesigning the Host and 
modem interface hardware. This left us with a de¬ 
cision between the Honeywell 516 and 316 computers. 
We chose the 316 on the basis of size and cost, feeling 
that the somewhat higher bandwidth of the 516 was 
not essential to the job and did not justify its higher 
price and the second cabinet which would have been 
required. 

TERMINAL IMP HARDWARE 

Figure 4 shows how the TIP fits the user into the 
network. Up to 63* terminals, either remote or local, 
of widely diverse characteristics may be connected to 
a given TIP and thereby “talk” into the network. It is 
also possible to connect a Host to a TIP in the usual 
way. 

The TIP is shown in Figure 5. It is built around a 
Honeywell H-316 computer with 20K (20,480 words) 
of core. It embodies a standard 16 port multiplexed 
memory channel with priority interrupts and includes 
a Teletype for debugging and program reloading. 
Other features of the standard IMP also present are 
a real-time clock, power-fail and auto-restart mecha¬ 
nisms, and a program-generated interrupt feature. 2 
As in the standard IMP, interfaces are provided for 
connecting to high-speed (50 kilobit, 230.4 kilobit, 
etc.) modems as well as to Hosts. The single-cabinet 
version limits the configuration to a total of three 
modem and/or Host interfaces, but an expansion 
cabinet may be used to increase this limit. More basic 
limits are set by the machine’s logical organization 
(specifically the number of available memory chan¬ 
nels) and the program bandwidth capability as dis¬ 
cussed below. 

Aside from the additional 8K of core memory, the 
primary hardware feature which distinguishes the 
TIP from a standard IMP is a Multi-Line Controller 
(MLC) which allows for connection of terminals to the 


Figure 5—Photograph of TIP 


* There ure G4 hardwure lines but line 0 is logically reserved by 
the program for special use. 
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IMP. Any of the MLC lines muy go to locnl terminals 
or via modems to remote terminals. As shown in 
Figure 6 the MLC consists of two portions, one a piece 
of central logic which handles the assembly and disas¬ 
sembly of characters and transfers them to and from 
memory, and the other a set of individual Line Inter¬ 
face Units (all identical except for small number of 
option jumpers) which synchronize reporting to in¬ 
dividual data bits between the central logic and the 
terminal devices and provide for control and status 
information to and from the modem or device. Line 
Interface Units may be physically incorporated one at 
a time as required. 

The MLC connects to the high-speed multiplexed 
memory channel option of the H-31G, and uses three 
of its channels as well as two priority interrupts and a 
small number of control instructions. The MLC is 
fabricated by BBN and is built from TTL integrated 
circuits. The MLC central controller, complete with 
power supply, is housed in an H-31C expansion chassis, 
and the entire MLC, as well as the computer itself, 
is mounted in a standard six-foot rack. Additional 
space is provided in the bottom of the rack for up to 
sixteen card-mounted modems of the Bell 103, 201, or 
202 variety, together with their power supplies. 

In order to accommodate a variety of devices, the 
controller handles a wide range of data rates and 
character sizes, in both synchronous and asynchronous 
modes. Data characters of length 5-, G-, 7-, or 8-bits 
are allowed by the controller. Since no interpretation 
of characters is done by the hardware, any character 
set, such as Baudot, Transcode ASCII or EBCDIC 
may be used. 

The following is a list of data rates accepted by the 
controller: 

ASYNCHRONOUS 


SYNCHRONOUS 


(Nominal Rates) 

Any rate up to and in¬ 

75 

1200 

cluding 19.2 Kb/s 

no 

1800 


134. 

5 2400 


150 

48001 


300 

9G00 [ output only 


600 

19200 


All above in bits/second 

The data format required of all devices is bit serial 
and each character indicates its own beginning by 
means of a start bit which precedes the data and in¬ 
cludes one or more stop bits at the end of the char¬ 
acter. This pcr-character framing is quite standard 
for asynchronous lines but synchronous lines, generally 
designed for higher bandwidths, frequently adopt 
some form of “binary synchronous communication” 


where the characters are packed tightly together into 
messages which arc then framed by special characters, 
framing is thereby amortized over the entire message, 
thus consuming a smaller fraction of the available 
bandwidth than the pcr-character framing which uses 
two or more bits for every character. The difficulties 
with this scheme, however, are that!'ft is more complex, 
requiring more sophisticated hardware at each end, 
and that no real standards exist which are adhered to 
by all or even most types of synchronous devices. We 
therefore decided to adopt pcr-character framing with 
start and stop bits even on synchronous lines. At a 
cost of some twenty percent of the bandwidth for 
framing, a very simple and general scheme is thus 
arrived at. A number of high speed terminal manu¬ 
facturers, faced with the same problems, have arrived 
at a similar conclusion. 

Given these characteristics, then, the controller will 
connect to the great majority of normal terminal de¬ 
vices such as Teletypes, alphanumeric CRT units, and 
modems, and also (with suitable remote interface 
units) to many peripheral devices such as card readers, 
line printers, and graphics terminals. Either full or 
half duplex devices can he accommodated. The stan¬ 
dard TIP program cannot deal with a magnetic tape 
unit through the MLC. However, as a special option, 
and with the use of additional core memory, standard 
Honeywell tape drives can be connected to the TIB 
as normal peripherals. 

The individual terminal line levels are consistent 
with EIA 11S-232C convention. Data rates and char¬ 
acter length are individually set for each line by the 
Ihe program. For incoming asynchronous lines, the 
program includes the capability for detecting char¬ 
acter length and line data rate as discussed below. 

Logically, the controller consists of G4 input ports 
and G4 output ports. Each input/output pair is brought 
out to a single connector which is normally connected 
to a single device. However, by using a special “Y” 
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Figure G—Block diagrum of TIP hardware 
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cable, the ports may go to completely separate devices 
of entirely different properties. Thus, input port 16 
may connect to a slow, asynchronous, 5-bit character 
keyboard while output port 16 connects to a high speed, 
synchronous display of some sort. In order to achieve 
this flexibility, the MLC stores information about each 
input and each output port and the program sets up 
this information for each half of each port in tum as 
it turns the ports “ON.” 

Several aspects of the MLC design are noteworthy. 
The central logic treats each of the 64 ports in suc¬ 
cession, each port getting 800 ns of attention per cycle. 
The port then waits the remainder of the cycle (51.2 
ns) for its next tum. For both input and output, two 
full characters are buffered in the central logic, the one 
currently being assembled or disassembled and one 
further character to allow for delays in memory ac¬ 
cessing. 

During input, characters from the various lines 
stream into a tumble table in memory on a first come, 
first served basis. Periodically a clock interrupt causes 
the program to switch tables and look for input. 

Output characters are fed to all lines from a single 
output table. Ordering the characters in this table in 
such a way as to keep a set of lines of widely diverse 
speeds solidly occupied is a difficult task. To assist the 
program in this, a novel mechanism has been built into 
the MLC hardware whereby each line, as it uses up a 
character from the output table, enters a request con¬ 
sisting of its line number into a “request” table in 
memory. This table is periodically inspected by the 
program and the requests are used in building the next 
output table with the characters in proper line se¬ 
quence. 

The design of the terminal interface portion of the 
MLC is modular. Each Line Interface Unit (LIU) 
contains all the logic required for full duplex, bit serial 
communication and consists of a basic bi-directional 
data section and a control and status section. The 
data section contains transmit and receive portions 
each with clock and data lines. For asynchronous 
devices the clock line is ignored and timing is pro¬ 
vided by the MLC itself. (For received asynchronous 
characters, timing is triggered by the leading edge of 
the start bit of each character.) 

The control and status monitor functions arc pro¬ 
vided for modems as required by the RS-232C specifi¬ 
cation. Four outputs are available for control functions 
and six inputs are available to monitor status. The 
outputs are under program control and are available 
for non-standard functions if the data terminal is not 
a modem. For example, these lines could be used to 
operate a local line printer. RS-232C connectors are 
mounted directly on the LIU cards. To allow for varia¬ 


tions in terminal and modem pin assignments, the 
signals arc brought to connector pins via jumpers on 
the card. 

The central MLC contains 256 ICs, many of which 
are MSI and some of which arc LSI circuits, and it is 
thus about the same complexity as the basic H-316. 
In addition each LIU contains 31 -ICs. A Terminal 
IMP including the MLC and with a typical interface 
configuration to high-speed circuits and Hosts is, order 
of magnitude, a $100,000 device. 

THE TERMINAL USER’S VIEW 

This section describes how a TIP user gains access 
to the network. The protocol described is of very 
recent origin and will undoubtedly change in response 
to usage which, as of this writing, is just commencing. 

In general a user must have some foreknowledge of 
the resource which he expects to access via the network. 
The TIP program implements a set of commands 8 for 
connecting to and disconnecting from remote sites, 
but once a terminal is connected to a particular system, 
the TIP becomes transparent to the conversation 
unless specially signalled. This is equivalent to a time¬ 
sharing system where the executive program is es¬ 
sentially out of the picture during periods while the 
user is dealing directly with his own set of programs. 

Because of the large number of different terminal 
types used in the network, the concept of the Network 
Virtual Terminal was developed. This is an imaginary 
but well-defined type of terminal. The TIP translates 
typed data to virtual terminal code before shipping it 
into the network, and conversely translates the remote 
system’s response back into the local terminal’s code. 
Thus, each Host system must deal only with this 
single terminal type. 

When the user at a terminal needs to talk directly 
to his TIP instead of to the remote Host, he issues a 
command which is distinguished by the fact that it 
always starts with the symbol @. One or more words, 
perhaps followed by a single parameter, then identify 
the type of command. 

Normally a user will go through four more or less 
distinct stages in typing into the net. First, he will 
be concerned with hardware, power, dialing in, etc. 
Then he will establish a dialogue with the TIP to get 
a comfortable set of parameters for his usage. Next, 
he will instruct the TIP to make a connection to a 
remote Host, and finally, he will mostly ignore the TIP 
as he talks to the remote Host. 

One of the more interesting features of the TIP is 
that it permits great flexibility in the types of ter¬ 
minals which may be attached to any port. This 
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TABLE II—Tip Commanda 


CLOSE 

close all outstanding connections 

HOST t 

focus attention on this host 0<if<256 

LOGIN 

start the initial connection procedure to get Telnet 
connections 
SEND BREAK 

send a Telnet break character 
SEND SYNC 

send a Telnet sync character and an INTERRUPT 
SENDER message 
ECHO ALL 

local TIP-generated echo—TIP echoes everything 
ECHO NONE 

remote Host-generated echo—TIP will echo commands 
ECHO HALFDUPLEX 

terminal-generated echo—TIP echoes nothing 

FLUSH 

dele+c all characters in input buffer 
TRANSMIT EVERY f 

sen' 1 off input buffer at least every #th character 
256 

TRANSMIT ON EVERY CHARACTER 
like TRANSMIT EVERY 1 
TRANSMIT ON LINEFEED 

send input buffer every time a linefeed encountered 
TRANSMIT ON MESSAGE-END 

send input buffer every time an EOM encountered 
TRANSMIT ON NO CHARACTER 

do not transmit on linefeed or EOM 


TRANSMIT NOW 

send off input buffer now 
DEVICE RATE # 


# is a 13 bit code specifying hardware rate and character 
size settings 


DEVICE CODE ASCII 
DEVICE CODE 2741 
DEVICE CODE EXECUPORT 
DEVICE CODE ODEC 


establish code 
conversion 


SEND TO HOST # 

RECEIVE FROM HOST # 
SEND TO SOCKET 4 
RECEIVE FROM SOCKET # 


establish parameters 
for manual initiation of 
connections 


PROTOCOL TO TRANSMIT 1 

PROTOCOL TO RECEIVE 'initiate connection 

PROTOCOL TO CLOSE TRANSMIT protocol manually 
PROTOCOL TO COSE RECEIVE 


PROTOCOL BOTH j abbreviations for 

PROTOCOL TO CLOSE BOTH ^simultaneous transmit 

jand receive protocols 

# GIVE BACK 

release control of captured device 
4 DIVERT OUTPUT 

capture device # and divert thiB terminal's output to it 
ABORT LOGIN 

abort the outstanding initial connection procedure 


flexibility presents a problem to the program which 
must determine what kind of terminal (speed, char¬ 
acter size, etc.) is attached to a given port before a 
sensible exchange of information is possible. To solve 
this problem, each terminal is assigned a special 
identifying character which the user, types repeatedly 
when starting to use a terminal. "When information 
starts to appear from a previously idle port, the TIP 
enters a “hunt” mode in which it interprets the ar¬ 
riving characters trying various combinations of 
terminal characteristics. All except the proper com¬ 
bination will cause the character to be garbled, pro¬ 
ducing something other than one of the legal terminal 
identifying characters. When the TIP thus identifies 
the correct set (which generally requires the user to 
repeat the identifier less than half a dozen times), 
it types out ‘HELLO in the terminal’s own language. 

In the next stage, the user initializes certain conver¬ 
sation parameters relating to message size and when 
to echo. He then establishes connection to a remote 
site using two commands which identify the desired 
remote site and the fact that the user wishes to be 
connected to the logger at that site. These commands 
are: 

@ HOST 15 
© LOGIN 

The LOGIN command actually sets in motion an 
elaborate exchange of messages between the TIP and 
the remote Host which normally result in a connection 
being made to that remote system. This command will 
be answered by an appropriate comment to the user 
indicating either that a connection has been made, that 
the remote site is not up, that it is up but actively re¬ 
fusing to converse, or that it is up but not responding 
enough even to refuse the connection. 

Once a connection is established, the user types 
directly to the logger. The TIP does not execute the 
actual login since this procedure varies from site to 
site. 

Throughout the user-to-Host dialogue, commands 
remain available at any time. Prior to closing the con¬ 
nection, the user must log out as required by the 
system he is using. He then gives the command 

@ CLOSE 

which causes the TIP to close the connection, in¬ 
forming the user when the process is finished. 

In addition to the above more or less standard pro¬ 
cedures, there are a number of less usual commands 
which set such things as device rate, character size, 
code types, etc. Such commands are used by a terminal 
on one port in setting parameters for a non-intcractive 
terminal, such as a printer or card reader, on some 
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other port. Other special commands permit conversa¬ 
tions directly between terminals on the same or dif¬ 
ferent TIPs, allow for binary mode, etc. Table II is 
a list of the commands with a brief explanation. For 
details refer to Reference 8. Figure 7 gives an example 
of typical dialogue. 

THE SOFTWARE 

Because the terminals connected to a TIP com¬ 
municate with Hosts at remote sites, the TIP, in ad¬ 
dition to performing the IMP function, also acts as 
intermediary between the terminal and the distant 
Host. This means that network standards for format 
and protocol must be implemented in the TIP. One 
can thus think of the TIP software as containing both 
a very simpleminded mini-Host and a regular IMP 
program. 

Figure 8 gives a simplified diagrammatic view of the 
program. The lower block marked “IMP” represents 
the usual IMP program. The two lines into and out 
of J .hat block are logically equivalent to input and 
output from a Host. The code conversion blocks are 
in fact surprisingly complex and include all of the 
material for dealing with diverse (and perverse) types 
of terminals. 

As the user types on the keyboard, characters go, 
via input code conversion, to the input block. Infor¬ 
mation for remote sites is formed into regular network 
messages and passes through the OR switch to the 
IMP program for transmittal. Command characters 
are fed off to the side to the command block where 
commands are decoded. The commands are then “per- 
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Figure 8—Block diagram of TIP program 


formed” in that they either set some appropriate param¬ 
eter or a flag which calls for later action. An example 
of this is the LOGIN command. Such a command in 
fact triggers a complex network protocol procedure, 
the various steps of which are performed by the PRO¬ 
TOCOL block working in conjunction with the remote 
Host through the IMPs. As part of this process an 
appropriate special message will be sent to the terminal 
via the Special Messages block indicating the status 
(success, failure, etc.) of the procedure. 

Once connection to a remote Host is established, 
regular messages flow directly through the Input block 
and on through the IMP program. Returning responses 
come in through the IMP, into the OUTPUT program 
where they are fed through the OUTPUT Code Con¬ 
version block to the terminal itself. 

Storage for the TIP program, including the standard 
IMP program, is just over 20K. This is roughly as 
follows: 


Standard IMP Program with buffers 


& tables 

12,000 

Special TIP code 

2,650 

Tables of Parameters 

1,800 

Buffer Storage for messages to and 

from terminals 

1,880 

Miscellaneous—I/O buffers, con- 

stants, etc. 

1,880 


PERFORMANCE 


Figure 7—Typical terminal user dialogue 


The program can handle approximately 100 kilobits 
of one-way terminal traffic provided the message size 
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is sufficiently large that per-message processing is 
unionized over many characters. Overhead per message 
is such that, if individual characters arc sent one at a 
time there is a loss of somewhere between a factor of 
ten and twenty in bandwidth. A different way to look 
at program performance is to observe that the per- 
character processing time is about 75 *is. 

These figures ignore the fact that the machine must 
devote some of its bandwidth to acting as an IMP, 
both for terminal traffic and for regular network 
traffic. About 5% of the machine is lost to acting as an 
IMP, even in the absence of traffic. If there is network 
traffic, more of the machine bandwidth is used up. 
Five hundred kilobits of two-way phone line and 
Host traffic saturates the machine without any ter¬ 
minal traffic*. 

In addition to bandwidth which goes into the IMP 
part of the job, another 10 percent of the (total) ma¬ 
chine is taken up simply in fielding clock interrupts 
from the Multi-Line Controller. This again is band¬ 
width used in idling even with no actual terminal 
traffic. 

The following formula summarizes, approximately, 
the bandwidth capabilities: 

P + H + 11T < 850 

where: 

P = total phone line traffic (in kilobits/sec) 
wherein, for example, a full duplex 50 Kb 
phone line counts as 100; 

H = total Host traffic (in kilobits/sec) wherein 
the usual full duplex Host interface, with 
its usual 10 fis/bit setting, counts as 200; 
and 

T = total terminal traffic (in kilobits/sec) wherein 
an ASCII terminal such as a (110-baud) 
full-duplex Teletype (ASR-33) counts as 
twice its baud rate (i.e., 0.220 Kb). 

This means that it takes eleven times as much 
program time to service every terminal character as 
it does to service a character’s worth of phone line 
or Host message. 

A further factor that influences terminal traffic 
handling capability has to do with the terminals 
themselves. Certain types of terminals require more 
attention from the program than others, independent 
of their speed but based rather on their complexity. 
In particular, for example, while an IBM 2741 nom- 


* The number 500 kilobits is for full size (8000 bit) messages. 
Shorter messages use up more capability per bit and thus reduce 
the overall bandwidth capability. 


inally runs at 134.5 hits per second, the complexity 
is such that it uses nearly three times the program 
bandwidth that, would be used in servicing a half- 
duplex ASCII terminal of equivalent speed. Allow¬ 
ances for such variations must be made in computing 
the machine's ability to service a particular configura¬ 
tion. It must, be borne in mind that'-all "of these per¬ 
formance figures arc approximations and that the 
actual rules are extremely complex, and will change 
as the program matures. 

DISCUSSION 

As the ARl’A Network grows, a number of areas 
of development seem likely to require attention. Cer¬ 
tain of these arc already pressing problems whereas 
others will begin to appear primarily as the network 
continues to grow and mature. 

Perhaps the most difficult aspect of a network such 
as this is the problem of reliability. A great deal of 
thought and effort has gone into trying to make the 
system reliable and network topology has been de¬ 
signed with the need for redundancy in mind. None¬ 
theless the problem of keeping a large number of 
computer systems going at widely separated sites is 
non-trivial. Am IMP’S mean-time-between-failure 
(MTBF) has been on the order of one month; con¬ 
siderably lower than expected. The problems arise 
from a variety of sources and the system is sufficiently 
complex that frequently the cause has been masked 
by the time the failure has been noted. Often the same 
failure will recur several times until the problem is 
identified and eliminated and this fact tends to decrease 
the apparent MTBF. In general a preponderance of 
the troubles stem from hardware failures, and we arc 
currently modifying several noisy and/or marginal 
portions of the I/O and interface hardware in hopes 
of obtaining significant improvement. 

Our strategy has generally been to spend time 
identifying the source of trouble, keeping the machine 
off the air if necessary, so that eventually the MTBF 
will increase. This has often meant, however, that 
a “down” was much longer than it would have been, 
had the foremost goal been to get the-machine running 
again immediately. With this strategy the average 
down time has been about 9 hours, giving rise to an 
average per-mnchinc down rate of about 2 percent. We 
hope to improve this situation in the near future by 
providing improved facilities for obtaining “program 
dumps” when a failure occurs. This will make it possible 
to bring machines back on the air almost, immediately 
in many cases, without jeopardizing valuable de¬ 
bugging data. 
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A word about our experience with the phone lines 
appears appropriate here as well. We have apparently 
been an unusual, if not unique, customer for the com¬ 
mon carriers in our degree of attention to line failures. 
Our ability to detect and report even brief outages has 
led through measured skepticism to eventual accept¬ 
ance by the carriers. In general, tests have indicated 
that the phone line error rates are about as predicted, 
i.e., approximately one in 10\ These occasional errors 
or error bursts do not appear to affect network per¬ 
formance adversely. The IMP-to-IMP error checking 
procedure has not, to our knowledge, admitted a 
single erroneous message. We have, however, had 
some difficulty with lines which were simply out of 
commission from time to time for extended periods 
(hours or even days). The reliability of the phone lines 
is roughly equivalent to the reliability of the IMPs, 
based on the number and duration of outages. Down 
times have decreased as the carriers have come to 
rr’cpt our complaints as generally legitimate. Overall 
the performance of the telephone equipment does not 
appear to constitute a problem in network growth. 

From a strictly technical viewpoint we view the in¬ 
corporation of higher bandwidth facilities as a natural 
and key part of network growth. While the present 
facilities arc not saturated by the present loads, we 
view this situation as a temporary one and something 
of a period of grace. As the network continues to grow 
over the next few years traffic can be expected to in¬ 
crease in a very non-linear fashion. Host protocol 
procedures (which have presented a sizable stumbling 
block to usage) are settling down so that software 
commitments can be made, and the advent of the 
Terminal IMP will bring an influx of new users who 
do not have the alternative of local resources. As a 
result we expect that traffic will begin to saturate 
some parts of the network. Terminal IMPs may well 
be called upon to service a larger number of terminals 
of higher bandwidth than can be handled by the 
present version. 

In anticipation of these requirements we are presently 
considering the design of a significantly faster and 
more modular version of the IMP. Its higher speed 
will permit it to take advantage of high speed (i.e., 
megabit and above) communication lines which the 
common carriers are currently developing. More 
generally it will be able to service high bandwidth 
requirements whenever and however they occur within 
the net. We arc currently studying a modular design 
which will permit connection of a greater number of 
interfaces than the present IMP can handle, While 
this work is only in the preliminary design stage, we 
feel that the interest and enthusiasm which have 
greeted the initial network suggest that it is not too 


early to consider ways to cope with growing traffic 
requirements. 

Another area of network development which has 
already received a great deal of attention and will 
require much more in the future is that of technical 
information interchange between the sites. To the 
user, the resources which arc.becoming available are 
staggering if not overwhelming. It is very difficult for 
an individual to discover what, if any, of the mass of 
programs that will be available through the network 
may be of interest or of use to him. To aid in this 
process a number of mechanisms are being implemented 
and we are cooperating closely with these efforts. In 
particular, at the Stanford Research Institute a Net¬ 
work Information Center (NIC) acts essentially as 
a library for documentation concerning the network. 
Here are maintained a number of pointer documents, 
specifically, (1) a Directory of Participants which lists 
critical personnel, phone numbers, etc., for each par¬ 
ticipating site, (2) a catalogue of the NIC Collection 
which lists the available documents indexed in a 
variety of ways, (3) a Resource Notebook which is a 
compendium that attempts to familiarize the reader 
with available program resources at each of the par¬ 
ticipating sites. 

Finally, there is the broad and exciting issue of how 
to cope with success. As the ARPA Network grows, 
and as diverse resources and users join the net, it is 
clear that a technology transfer must occur; the net¬ 
work probably must shift from a government-sup¬ 
ported research and development activity to an 
ongoing national service of some kind. However, the 
computer-communications environment in the U.S. 
is rather complex, and is populated with many legal, 
economic, and political constraints. Within this en¬ 
vironment, it is not easy to perform the technology 
transfer, and many groups, including ARPA and BBN, 
have been considering possible alternative plans. We 
are optimistic that a way will be found to provide a 
suitable legal and political base for expansion of the 
present ARPA Network into a widely useful public 
communication system. 
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INTRODUCTION 

The ARPA Network (ARPANET) project brought 
together many individuals with diverse backgrounds, 
philosophies, and technical approaches from the fields 
of computer science, communication theory, operations 
research and others. The project was aimed at providing 
an efficient and reliable computer communications 
system (using message switching techniques) in which 
computer resources such as programs, data, storage, 
special purpose hardware etc., could be shared among 
computers and among many users. 38 The variety of 
design methods, ranging from theoretical modeling to 
hardware development, were primarily employed 
independently, although cooperative efforts among 
designers occurred on occasion. As of November, 1971, 
the network has been an operational facility for many 
months, with about 20 participating sites, a network 
information center accessible via the net, and well over 
a hundred researchers, system programmers, computer 
center directors and other technical and administrative 
personnel involved in its operation. 

In this paper, we review and evaluate the methods 
used in the ARPANET design from the vantage of 
over two years’ experience in the development of the 
network. In writing this paper, the authors have each 
made equal contributions during a series of intensive 
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DAlfC 15-7O-C-O120 at tho Network Analysis Corporation, 
Contract Nos. DAHC 1W39-C-0179 and DAHC-71-C-0088 at 
Bolt Beranek and Newman Inc., and Contract No. DAHC 
15-09-C-0285 at the University of California at Los Angeles. 


discussions and debates. Rather than present merely a 
summary of the procedures that were used in the 
network design, we have attempted to evaluate each 
other’s methods to determine their advantages and 
drawbacks. Our approaches and philosophies have often 
differed radically and, as a result, this has not been an 
easy or undisturbing process. On the other hand, we 
have found our collaboration to be extremely rewarding 
and, notably, we have arrived at many similar con¬ 
clusions about the network’s behavior that seem to be 
generally applicable to message switched networks. 

The essence of a network is its design philosophy, its 
performance characteristics, and its cost of implementa¬ 
tion and operation. Unfortunately, there is no generally 
accepted definition of an “optimal” network or <.”en o f 
a “good” network. For example, a network designed to 
transmit large amounts of data only during late evening 
hours might call for structural and performance char¬ 
acteristics far di fferent from one servicing large numbers 
of users who are rapidly exchanging short messages 
during business hours. We expect this topic, and others 
such as the merits of message switching vs. circuit 
switching or distributed vs. centralized control to be a 
subject of discussion for many years. i.u.m.jj.h.n 

A cost analysis performed in 1967-1968 for the ARPA 
Network indicated that the use of message switching 
would lead to more economical communications and 
better overall availability and utilization of resources 
than other methods. 34 ’ 38 In addition to its impact on 
the availability of computer resources, this decision has 
generated widespread interest in store-and-forward 
communications. In many instances, the use of store- 
and-forward communication techniques can result in 


255 


25G Spring Joint Computer Conference, 1972 


greater flexibility, higher reliability, .significant tech¬ 
nical advantage, and substantial economic savings over 
the use of conventional common carrier offerings. An 
obvious trend toward increased computer and com¬ 
munication interaction has begun. In addition to the 
ARPANET, research in several laboratories is under 
way, small experimental networks are being built, and 
a few examples of other government, and commercial 
not.works are already apparont.‘ ,, - 3l - 40,41, "- < * ,M 

In the ARPANET, each time-sharing or hutch 
processing computer, ctdled a Host., is connected to a 
small computer culled an Interface Message Processor 
(IMP). The IMPs, which arc interconnected by leased 
50 kilobit/second circuits,' handle all network com¬ 
munication for their Hosts. To send a message to 
another Host, a Host precedes the text, of its message 
with an address and simply delivers it to its IMP. The 
IMPs then determine the route, provide error control, 
und notify the sender of its receipt. The collection of 
Hosts, IMPs, and circuits forms the message switched 
resource ‘'haring network. A good description of the 
ARPA..->, , and some curly results on protocol develop¬ 

ment and modeling are given in References 3, 12, 15, 
23 and 38. Some experimental utilization of the 
ARPANET is described in Reference 42. A more recent 
evaluation of such networks and u forward look is 
given in References 35 and 39. 

The development of the Network involved four 
principal activities: 

(1) The design of the IMPs to act as nodal storc- 
and-forward switches, 

(2) The topological design to specify the capacity 
and location of each communication circuit 
within the network, 

(3) The design of higher level protocols for the use 
of the network by time-sharing, batch pro¬ 
cessing and other data processing systems, and 

(4) System modeling and measurement of network 
performance. 

Each of the first three activities were essentially per¬ 
formed independently of each other, whereas the 
modeling effort partly affected the IMP design effort, 
and closely interacted with the topological design 
project. 

The IMPs were designed by Bolt. Beranek and 
Newman Inc. (BBN) and were built to operate in¬ 
dependent. of the exact network connectivity; the 
topological structure was specified by Network Analysis 
Corporation (NAC) using models of network per¬ 
formance developed by NAC and by the University of 
California at Los Angeles (UCLA). The major efforts 
in the urea of system modeling were 'performed at 


UCLA using theoretical and simulation tccimwuics 
Network perfnriiiunee measurements have hern con¬ 
ducted during the development of the network by 
BBN and by the Network Measurement Center at 
UCLA. To facilitate effective- use of the net, higher 
level (user) protocols arc under development by a 
group of representatives of universities and research 
centers. This group, known as the Network Working 
Group, has already specified a Host to Host protocol 
und a Telnet protocol, and is in the process of com¬ 
pleting other function oriented protocols. , - ,u We make 
no attempt to elaborate on the Host to Host, protocol 
design problems in this paper. 

THE NETWORK DESIGN PROBLEM 

A variety of performance requirements and system 
constraints were considered in the design of the net. 
Unfortunately, many of the key design objectives had 
to bo specified long before the actual user requirements 
couid be known. Once the decision to employ message 
switching was made, and fifty kilobit/second circuits 
were chosen, the critical design variables were the 
network operating procedure and the network topology; 
the desired values of throughput, delay, reliability and 
cost were system performance and constraint variables. 
Other constraints affected the structure of the network, 
but not its overall properties, such as those arising from 
decisions about the length of time a message could 
remain within the network, the location of IMPs 
relative to location of Hosts, and the number of Hosts to 
be handled by a single IMP. 

In this section, wc identify the central issues related 
to IMP design, topological design, and network 
modeling. In the remainder of the paper, we describe 
the major design techniques which have evolved. 

IMP properties 

The key issue in the design of the IMPs was the 
definition of a relationship between the IMP subnet 
and the Hosts to partition responsibilities so that 
reliable and efficient operation would be achieved. The 
decision was made to build an autonomous subnet, 
independent (as much as possible) of the operation of 
any Host. The subnet was designed to function as a 
"communications system’’; issues concerning the use of 
the subnet by the Hosts (such as protocol development) 
were initially left to the Hosts. For reliability, the 
l.MPs were designed to be robust against all line failures 
and the vast majority of IMP and Host failures. This 
decision required routing strategies that dynamically 
adapt to changes in the states of IMPs and circuits. 
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and an elaborate flow control strategy to protect the 
subnet against Host malfunction and congestion due to 
IMP buffer limitations. In addition, a statistics and 
status reporting mechanism was needed to monitor the 
behavior of the network. 

The number of circuits that an IMP must handle is a 
design constraint directly affecting both the structure 
of the IMP and the topological design. The speed of the 
IMP and the required storage for program and buffers 
depend directly upon the total required processing 
capacity, which must be high enough to switch traffic 
from one line to another when all are fully occupied. Of 
great importance is the property that all IMPs operate 
with identical programs. This technique greatly 
simplifies the problem of network planning and main¬ 
tenance and makes network modifications easy to 
perform. 

The detailed physical structure of the IMP is not 
discussed in this paper. 211 However, the operating 
procedure, which guides packets through the net, is 
very much of interest here. The flow control, routing, 
and error control techniques are integral parts of the 
operating procedure and can be studied apart from the 
hardware by which they are implemented. Most 
hardware modifications require changes to many 
IMPs already installed in the field, while a change in 
the operating procedure can often be made more 
conveniently by a change to the single operating 
program common to all IMPs, which can then be 
propagated from a single location via the net. 

Topological properties 

The topological design resulted in the specification of 
the location and capacity of all circuits in the network. 
Projected Host—Host traffic estimates were known at 
the start to be either unreliable or wrong. Therefore, 
the network was designed under the assumption of 
equal traffic between all pairs of nodes. (Additional 
superimposed traffic was sometimes included for those 
nodes with expectation of higher traffic requirements.) 
The topological structure was determined with the aid 
of specially developed heuristic programs to achieve a 
low cost, reliable network with a high throughput and 
a general insensitivity to the exact traffic distribution. 
Currently, only 50 kilobit/second circuits arc being 
used in the ARPANET. This speed line was chosen to 
allow rapid transmission of short, messages for inter¬ 
active processing (e.g., less than 0.2 seconds average 
packet delay) as well as to achieve high throughput 
(e.g., at least 50 kilobits/second) for transmission of 
long messages. For reliability, the network was con¬ 
strained to have at least two independent paths between 
each pair of IMPs. 


The topological design problem requires consideration 
of the following two questions: 

(1) Starting with a given state of the network 
topology, what circuit modifications are required 
to add or delete a set of IMPs? 

(2) Starting with a given state of'network topology, 
when and where should circuits be added or 
deleted to account for long term changes in 
network traffic? 

If the locations of all network nodes are known in 
advance, it is clearly most efficient to design the 
topological structure as a single global effort. However, 
in the ARPANET, as in most actual networks, the 
initial designation of node locations is modified on 
numerous occasions. On each such occasion, the 
topology can be completely reoptimized to determine a 
new set of circuit locations. 

In practice, there is a long lead time between the 
ordering and the delivery of a circuit, and major topo¬ 
logical modifications cannot be made without sub¬ 
stantial difficulty. It is therefore prudent to add or 
delete nodes with as little disturbance as possible to 
the basic network structure consistent with overall 
economical operation. Figure 1 shows the evolution of 
the ARPANET from the basic four IMP design in 19G9 
to the presently planned 27 IMP version. Inspection of 
the 24 and 27 IMP network designs reveals a few 
substantial changes in topology that take advantage of 
the new nodes being added. Surprisingly enough, a 
complete “reoptimization” of the 27 IMP topology 
yields a network only slightly less expensive (about 
1 percent) than the present network design. 28 

Network models 

The development of an accurate mathematical model 
for the evaluation of time delay in computer networks is 
among the more difficult of the topics discussed in this 
paper. On the one hand, the model must properly 
reflect the relevant features of the network structure 
and operation, including practical constraints. On the 
other hand, the model must result in a mathematical 
formulation which is tractable and from which mean¬ 
ingful results can be extracted. However, the two 
requirements are often incompatible and we scarcli for 
an acceptable compromise between these two extremes. 

The major modeling effort thus far has been the study 
of the behavior of networks of queues. 21 This emphasis 
is logical since in message switched systems, messages 
experience queueing delays as they pass from node to 
node and thus a significant performance measure is the 
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Figure 1—The evolution of the ARPANET 


speed at which messages can be delivered. The queueing 
models were developed at a time when there were no 
operational networks available for experimentation and 
model validation, and simulation was the only tool 
capable of testing their validity. The models, which at 
all times were recognized to be idealized statements 
about the real network, were nonetheless crucial to the 
ARPANET topological design effort since they afforded 
the only known way to quantitatively predict the 
properties of different routing schemes and topological 
structures. The models have been subsequently demon¬ 
strated to be very accurate predictors of network 
throughput and indispensable in providing analytical 
insight into the network’s behavior. 

The key to the successful development of tractable 
models has been to factor the problem into a set of 
simpler queueing problems. There arc also heuristic 
design procedures that one can use in this case. These 
procedures seem to work quite well and are described 
later in the paper. However, if one specializes the 
problem and removes some of the real constraints, 
theory and analysis become useful to provide under¬ 
standing, intuition and design guidelines for the. original 
constrained problem. This approach uncovers global 
properties of network behavior, which provide keys to 


good heuristic design procedures and ideal performance 
bounds. 


DESIGN TECHNIQUES 

In this section we describe the approaches taken to 
the design problems introduced in the previous section. 
We first summarize the important properties of the 
ARPANET design: 

(1) A communications cost of less than 30 cents per 
thousand packets (approximately a megabit). 

(2) Average packet delays under 0.2 seconds through 
the net. 

(3) Capacity for expansion to 64 IMPs without 
major hardware or software redesign. 

(4) Average total throughput capability of 10-15 
kilobits/sccond for all Hosts at an IMP. 

(5) Peak throughput capability of 85 kilobits/second 
per pair of IMPs in an otherwise unloaded 
network. 

(6) Transparent communications with maximum 
message size of approximately 8000 bits and 
error rates of one bit in 10 15 or less. 
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(7) Approximately 98 percent availability of any 
IMP and close to 100 percent availability of all 
operating IMPs from any operable IMP. 

The relationships between the various design efforts 
are illustrated by these properties. The topological 
design provides for both a desired average throughput 
and for two or more paths to be fully used for com¬ 
munication between any pair of Hosts. The operating 
procedure should allow any pair of Hosts to achieve 
those objectives. The availability of IMPs to com¬ 
municate reflects both the fact that IMPs are down 
about 2 percent of the. time and that the topology is 
selected so that circuit failures contribute little addi¬ 
tional to the total system downtime. 

IMP design 

Jhc IMP design consists of two closely coupled but 
nonetheless separable pieces—the physical hardware 
specification (based on timing and reliability considera¬ 
tions and the operating procedure) and the design and 
implementation of the operating procedure using the 
specified IMP hardware. The IMP originally developed 
for the ARPANET contains a 16-bit one microsecond 
computer that can handle a total of about % megabits/ 
second of "useful” information on a total of approxi¬ 
mately one megabit/second of circuit capacity (e.g., 
twenty 50 kilobit/second circuits). Hardware is likely 
to change as a function of the required IMP capacity 
but an operating procedure that operates well at one 
IMP capacity is likely to be transferable to machines 
that provide different capacity. However, as a network 
grows in size and utilization, a more comprehensive 
operating procedure that takes account of known 
structural properties, such as a hierarchical topology, 
is appropriate. 

Four primary areas of IMP design, namely message 
handling and buffering, error control, flow control, and 
routing are discussed in this section. The IMP provides 
buffering to handle messages for its Host and packets 
for other IMPs. Error control is required to provide 
reliable communication of Host messages in the 
presence of noisy communication circuits. The design 
of the operating procedure should allow high through¬ 
put in the net under heavy traffic loads. Two potential 
obstacles to achieving this objective are: (1) The net 
can become congested and cause the throughput to 
decrease with increasing load, and (2) The routing 
procedure may be unable to always adapt sufficiently 
fast to the rapid movement of packets to insure efficient 
routing. A flow control and routing procedure is 
needed that can efficiently meet this requirement. 


Message handling and buffering 

In the ARPANET, the maximum message size was 
constrained to be approximately 8000 bits. A pair of 
Hosts will typically communicate over the net via a 
sequence of transmitted messages. To obtain delays of 
a few tenths of a second for such" messages and to lower 
the required IMP buffer storage, the IMP program 
partitions each message into one or more packets each 
containing at most approximately 1000 bits. Each 
packet of a message is transmitted independently to 
the destination where the message is reassembled by 
the IMP before shipment to that destination Host. 
Alternately, the Hosts could assume the responsibility 
for reassembling messages. For an asynchronous IMP- 
Host channel, this marginally simplifies the IMP's 
task. However, if every IMP-Host channel were syn¬ 
chronous, and the Host provided the reassembly, the 
IMP task can be further simplified. In this latter case, 
"IMP-likc” software would have to be provided in 
each Host. 

The method of handling buffers should be simple to 
allow for fast processing and a small amount of program. 
The number of buffers should be sufficient to store 
enough packets for the circuits to be used to capacity; 
the size of the buffers may be intuitively selected with 
the aid of simple analytical techniques. For example, 
fixed buffer sizes arc useful in the IMP for simplicity 
of design and speed of operation, but inefficient utiliza¬ 
tion can arise because of variable length packets. If 
each buffer contains A words of overhead and provides 
space for M words of text, and if message sizes arc 
uniformly distributed between 1 and L, it can be 
shown" that the choice of M that minimizes the ex¬ 
pected storage is approximately \/ AL. In practice, M 
is chosen to be somewhat smaller on the assumption 
that most traffic will be short and that the amount of 
overhead can be as much as, say, 25 percent of buffer 
storage. 

Error control 

The IMPs must assume the responsibility for pro¬ 
viding error control. There are “four possibilities to 
consider: 

(1) Messages arc delivered to their destination out 
of order. 

(2) Duplicate messages are delivered to the 
destination. 

(3) Messages are delivered with errors. 

(4) Messages are not delivered. 
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The tik.sk of proper sequencing of messages for 
delivery to the destination Host actually falls in the 
province; of both error control and flow control. If at 
most, one message at a time is allowed in the net between 
a pair of Hosts, proper sequencing occurs naturally. A 
duplicate {nickel will arrive at the destination IMP 
after an acknowledgment has been missed, thus causing 
a successfully received packet to be retransmitted. The 
IMPs can handle; the first two conditions by assigning 
a sequence number to each packet as it enters the 
network and processing the sequence number at the 
destination IMP. A Host that performs reassembly can 
also assign and process sequence numbers and check 
for duplicate packets. For many applications, the order 
of delivery to the destination is immaterial. For priority 
messages, however, it is typically the case that fast 
delivery requires a perturbation to the sequence. 

Errors arc primarily caused by noise on the. com¬ 
munication circuits and are handled most simply by 
error detection and retransmission between each pair of 
IMPs along the transmission path. This technique 
requires extra storage in the IMP if either circuit 
speeds or circuit lengths substantially increase. Failures 
in detecting errors can be made to occur on the order of 
years to centuries apart with little extra overhead 
(20-30 parity bits per packet with the 50 kilobit/second 
circuits in the ARPANET). Standard cyclic error 
detection codes have been usefully applied here. 

A reliable system design insures that each trans¬ 
mitted message is accurately delivered to its intended 
destination. The occasional time when an IMP fails and 
destroys a useful in-transit message is likely to occur 
far less often than a similar failure in the Hosts and has 
proven to be unimportant in practice, as are errors due 
to IMP memory failures. A simple end to end retrans¬ 
mission strategy will protect against these situations, 
if the practical need should arise. However, the IMPs 
arc designed so that they can be removed from the 
network without destroying their internally stored 
packets. 

Flow control 

A network in which packets may freely enter and 
leave can become congested or logically deadlocked and 
cause the movement of traffic to halt. 1 ' 17 Flow control 
techniques are required to prevent these conditions 
from occurring. The provision of extra buffer storage 
will mitigate against congestion and deadlocks, but 
cannot in general prevent them. 

The sustained failure of a destination Host to accept 
packets from its IMP at the rate of arrival will cause the 
net to fill up and become congested. Two kinds of 


logical deadlocks, known as reassembly lockup and 
store-und-forward lockup may also occur. In reassembly 
lockup, the remaining packets of partially reassembled 
messages are blocked from reaching the destination 
IMP (thus preventing the message from being com¬ 
pleted and the reassembly space freed J by other 
packets in the net that are waiting for-reassembly space 
at that destination to become free. In a store-and- 
forward lockup, the destination has room to accept 
arriving packets, but the packets interfere with each 
other by tying up buffers ill transit in such a way that 
none of the packets are able to reach the destination.” 
These phenomena have only been made to occur during 
very carefully arranged testing of the ARPANET and 
by simulation . ,,J 

In the original ARPANET design, the use of soft¬ 
ware links and RFNMS protected against congestion 
by a single link or a small set of links. However, the 
combined traffic on a large number of links could still 
produce congestion. Although this strategy did not 
protect against lockup, the method has provided ample 
protection for the levels of traffic encountered by the 
net to date. 

A particularly simple flow control algorithm that 
augments the original IMP design to prevent congestion 
and lockup is also described in Reference 17. This 
scheme includes a mechanism whereby packets may be 
discarded from the net at the destination IMP when 
congestion is about to occur, with a copy of each dis¬ 
carded packet to be retransmitted a short time later by 
the; originating Host’s IMP. Rather than experience 
excessive delays within the net as traffic levels are 
increased, the traffic is queued outside; the net so that 
the transit time delays internal to the net continue to 
remain small. This strategy prevents the; insertion of 
more traffic into the net than it can handle. 

It is important to note the dual requirement for small 
delays for interactive traffic and high bandwidth for the 
fast transfer of files. To allow high bandwidth between 
a pair of Hosts, the net must be able to accept a steady 
flow of packets from one Host and at the same time; be 
able to rapidly quench the flow at the entrance to the 
source; IMP in the; event of imminent congestion at the 
destination. This usually requires that a separate 
provision be made' in the algorithm to -protect short 
interactive; messages from experiencing unnecessarily 
high delays. 

Routing 

Network routing strategies for distributed networks 
require routing decisions to be made with only in¬ 
formation available to an IMP and the IMP must 
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execute those decisions to effect the routing.A 
simple example of such a strategy is to have each IMP 
handling a packet independently route it along its 
current, estimate of the shortest path to the destination, 

For many applications, it suffices to deal with an 
idealized routing strategy which may not simulate the 
IMP routing functions in detail or which uses informa¬ 
tion not available to the IMP. The general properties 
of both strategies arc usually similar, differing mainly 
in certain implementation details such as the avail¬ 
ability of buffers or the constraint of counters and the 
need for the routing to quickly adapt to changes in 
IMP and circuit status. 

The IMPs perform the routing computations using 
information received from other IMPs and local 
information such as the alive/dead state of its circuits. 
In the normal case of time varying loads, local informa¬ 
tion alone, such as the length of internal queues, is 
insufficient to provide an efficient routing strategy 
without assistance from the neighboring IMPs. It is 
possible to obtain sufficient information from the 
r. eight' .;rs to provide efficient routing, with a small 
amount of computation needed per IMP and without 
each IMP requiring a topological map of the network. 
In certain applications where traffic patterns exhibit 
regularity, the use of a central controller might be 
preferable. However, for most applications which 
involve dynamically varying traffic flow, it appears 
that a central controller cannot be used more effectively 
than the IMPs to update routing tables if such a 
controller is constrained to derive its information via 
the net. It is also a less reliable approach to routing 
than to distribute the routing decisions among the 
IMPs. 

The routing information cannot be propagated about 
the net in sufficient time to accurately characterize the 
instantaneous traffic flow. An efficient algorithm, there¬ 
fore, should not focus oil the movement of individual 
packets, but rather use topological or statistical in¬ 
formation in the selection of routes. For example, by 
using an averaging procedure, the flow of traffic can be 
made to build up smoothly. This allows the routing 
algorithm ample time to adjust its tables in each IMP 
in advance of the build-up of traffic. 

The scheme originally used in the ARPA network 
had each IMP select one output line per destination 
onto which to route packets. The line was chosen to be 
the one with minimum estimated time delay to the 
destination. The selection was updated every half 
second using minimum time estimates from the neigh¬ 
boring IMPs and internal estimates of the delay to each 
of the neighbors. Even though the routing algorithm 
only selects one line at a time per destination, two 
output lines will be used if a queue of packets waiting 


transmission on one line builds up before the routing 
update occurs and another line is chosen. Modifications 
to the scheme which allow several lines per destination 
to be used in an update interval (during which the 
routing is not changed) arc possible using two or more 
time delay estimates to select the : paths. 

In practice, this approach has worked quite effectively 
with the moderate levels of traffic experienced in the 
net. For heavy traffic flow, this strategy may be 
inefficient, since the routing information is based on 
the length of queues, which wc have seen can change 
much faster than the information about the change can 
be distributed. Fortunately, this information is still 
usable, although it can be substantially out of date and 
will not, in general, be helpful in making efficient 
routing decisions in the heavy traffic case. 

A more intricate scheme, recently developed by 
BBN, allows multiple paths to be efficiently used even 
during heavy traffic. 18 Preliminary simulation studies 
indicate that it can be tailored to provide efficient 
routing in a network with a variety of heavy traffic 
conditions. This method separates the problem of 
defining routes onto which packets may be routed from 
the problem of selecting a route when a particular 
packet must be routed. By this technique, it is possible 
to send packets down a path with the fewest IMPs and 
excess capacity, or when that path is filled, the one with 
the next fewest IMPs and excess capacity, etc. 

A similar approach to routing was independently 
derived by NAC using an idealized method that did not 
require the IMPs to participate in the routing decisions. 
Another approach using a flow deviation technique has 
recently been under study at UCLA. 13 The intricacies of 
the exact approach lead to a metering procedure that 
allows the overall network flow' to be changed slowly for 
stability and to perturb existing flow' patterns to obtain 
an increased flow. These approaches all possess, in 
common, essential ingredients of a desirable routing 
strategy. 

Topological considerations 

An efficient topological design provides a high 
throughput for a given cost. Although many measures 
of throughput are possible, a convenient one is the ’ 
average amount of traffic that a single IMP can send 
into the network when all other IMPs are transmitting 
according to a specified traffic pattern. Often, it is 
assumed that all other IMPs are behaving identically 
and each IMP is sending equal amounts of traffic to 
each other IMP. The constraints on the topological 
design are the available common carrier circuits, the 
target cost or throughput, the desired reliability, and 
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TABLE 1—23 Node 28 Link ARPA 
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Circuits 
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28 

1 

1 

27 

28 

28 

26 

378 

378 

2 5 

3276 

3276 

24 

20475 

20475 

23 

98280 

98280 

22 

376740 

376740 

21 

1184040 

1184040 

20 

3108105 

3108105 

19 

6906900 

6906900 

18 

13123110 | 

j 13123110 

17 

21474180 t 

> 21474180 

16 

30421755 , 

30421755 

1.6 

37442160 

37442160 

14 

40116600 

40116600 

13 

37442160 

37442160 

id 

30421755 

30421755 

11 

21474180 

21474180 

10 

13123110 

13123110 

9 

6906900 

69Q6900 

8 

3108108 

3108108 

7 

1184040 1 

1184040 

6 

376740 

349618 

5 

98280 

= 70547 

4 
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3 
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2 
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30 

1 

28 
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the cost of computation required to perform the 
topological design. 

Since, there was no clear specification of the amount 
of traffic that the network would have to accommodate 
initially, it was first constructed with enough excess 
capacity to accommodate any reasonable traffic require¬ 
ments. Then as new IMPs were added to the system, 
the capacity was and is still being systematically 
reduced until the traffic level occupies a substantial 
fraction of the network's total capacity. At this point, 
the net’s capacity will be increased to maintain the 
desired percentage of loading. At the initial stages of 
network design, the “two-connected” reliability con¬ 
straint essentially determined a minimum value of 
maximum throughput. This constraint forces the 
average throughput to be in the range 10-15 kilobits per 
second per IMP, when 50 kilobit/sec circuits are used 
throughout the network, since two communication 
paths between every pair of IMPs are needed. Alterna¬ 
tively, if this level of throughput is required, then the 
reliability specification of “two-connectivity” can be 
obtained without additional cost. 


Reliability computations 

A simple and natural characterization of network 
reliability is the ability of the network to sustain 
communication between all operable pairs of IMPs. For 
design purposes, the requirement o/ two. independent 
paths between nodes insures that at least two IMPs 
and/or circuits must fail before any pair of operable 
IMPs cannot communicate. This criterion is independent 
of the properties of the IMPs and circuits, docs not take 
into account the degree” of disruption that may occur 
and hence, docs not reflect the actual availability of 
resources in the network. A more meaningful measure 
is the average fraction of IMP pairs that cannet com¬ 
municate because of IMP and circuit failures. This 
calculation requires knowledge of the IMP and circuit 
failure rates, and could not be performed until enough 
operating data was gathered to make valid predictions. 

To calculate network reliability, wc must consider 
elementary network structures known as cutsets. A 



figure 2 Network availability vs. IMP and circuit reliability 
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cutset is a set of circuits and/or IMPs whose removal 
from the network breaks all communication paths 
between at least two operable IMPs. To calculate 
reliability, it is often the case that all cutsets must be 
either (•numerated or estimated. As an example, in a 
23 IMP, 28 circuit ARPA Network design similar to 
the one shown in Figure 1(d), there arc over twenty 
million ways of deleting only circuits so that the 
remaining network has at least one operable pair of 
IMPs with no intact communication paths. Table 1 
indicates the numbers of cutsets in the 23 IMP network 
as a function of the number of circuits they contain. 

A combination of analysis and simulation can be 
used to compute the average fraction of non-com¬ 
municating IMP pairs. Detailed descriptions of the 
analysis methods arc given in Reference 44 while their 
application to the analysis of the ARPANET is dis¬ 
cussed in Reference 43. The results of an analysis of 
the 23 IMP version of the network arc shown in Figure 
2. The curve marked A shows the results under the 
assumption that IMPs do not fail, while the curve 
marked B shows the case where circuits do not fail. 
The curve marked C assumes that both IMPs and 
circuits fail with equal probability. In actual operation, 
the average failure probability of both IMPs and 
circuits is about 0.02. For this value, it can be seen that 
the effect of circuit failures is far less significant than 
the effect of IMP failures. If an IMP fails in a network 
with n IMPs, at least n— 1 other IMPs cannot com¬ 
municate with it. Thus, good network design cannot 
improve upon the effect directly due to IMP failures, 
which in the ARPANET is the major factor affecting 
the reliability of the communications. Further, more 
intricate reliability analyses which consider the loss of 
throughput capacity because of circuit failures have 
also been performed and these losses have been shown 
to be negligible. 28 Finally, unequal failure rates due to 
differences in line lengths have been shown to have 
only minor effects on the analysis and can usually be 
neglected . 77 

Topological optimization 

During the computer optimization process, the 
reliability of the topology is assumed to be acceptable if 
the network is at least two-connccted. The object of 
the optimization is to decrease the ratio of cost to 
throughput subject to an overall cost limitation. This 
technique employs a sophisticated network optimization 
program that utilizes circuit exchange heuristics, 
routing and flow analysis algorithms, to generate low 
cost designs. In addition, two time delay models were 
initially used to (1) calculate the throughput corre¬ 


sponding to an average time delay of 0.2 seconds, 
(2) estimate the packet rejection rate due to all buffers 
filling at an IMP. As experience with these models 
grew, the packet rejection rate was found to be negligible 
and the computation discontinued. The delay computa¬ 
tion (Equation (7) in a later section) was subsequently 
first replaced by a hcuriatic calculation to speed the 
computation and later eliminated after it was found 
that time delays could be guaranteed to be acceptably 
low by preventing cutsets from being saturated. This 
“threshold” behavior is discussed further in the next 
section. 

The basic method of optimization was described in 
Reference 12 while extensions to the design of large 
networks arc discussed in Reference 9. The method 
operates by initially generating, either manually or by 
computer, a “starting network” that satisfies the overall 
network constraints but is not, in general, a low cost 
network. The computer then iteratively modifies the 
starting network in simple steps until a lower cost 
network is found that satisfies the constraints or the 
process is terminated. The process is repeated until no 
further improvements can be found. Using a different 
starting network can result in a different solution. 
However, by incorporating sensible heuristics and by 
using a variety of carefully chosen starting networks and 
some degree of man-machine interaction, “excellent” 
final networks usually result. Experience has shown 
that there arc a wide variety of such networks with 
different topological structures but similar cost and 
performance. 

The key to this design effort is the heuristic procedure 
by which the iterative network modifications arc made. 
The method used in the ARPANET design involves the 
removal and addition of one or two circuits at a time. 
Many methods have been employed, at various times, 
to identify the appropriate circuits for potential addi¬ 
tion or deletion. For example, to delete uneconomical 
circuits a straightforward procedure simply deletes 
single circuits in numerical order, reroutes traffic and 
reevaluates cost until a decrease in cost per megabit is 
found. At this point, the deletion is made permanent 
and the process begins again. A somewhat more 
sophisticated procedure deletes circuits in order of 
increasing utilization, while a unore complex method 
attempts to evaluate the effect of the removal of any 
circuit before any deletion is attempted. The circuit 
with the greatest likelihood of an improvement is then 
considered for removal and so on. 

There are a huge, number of reasonable heuristics for 
circuit exchanges. After a great deal of experimentation 
with many of these, it appears that the choice of a 
particular heuristic is not critical. Instead, the speed 
and efficiency with which potential exchanges can be 
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investigat.ee! appears to be tbe limiting factor affecting 
the quality of the final design. Finally, as the .size of 
the. network increases, the greater the cost becomes to 
perform any circuit exchange optimization. Decom¬ 
position of the network design into regions becomes 
necessary and additional heuristics are needed to 
determine effective decompositions. It presently appears 
that these methods can be used to design relatively 
efficient networks with a few hundred I.MPs while 
substantially new procedures will be necessary for 
networks of greater size. 

The topological design requires a routing algorithm 
to evaluate the throughput capability of any given 
network. Its properties must reflect those of an im- 
plementable routing algorithm, for example, within 
the ARPANET. Although the routing problem can be 
formulated as a “multicommodity flow problem" 10 and 
solved by linear programming for networks with 20-30 
IMPs, 8 faster techniques are needed when the routing 
a! Thm is incorporated in a design procedure. The 
design procedure, for the ARPA Network topology 
iter r *"'cly analyzes thousands of networks. To satisfy 
the requirements for speed, an algorithm which 
selects the least utilized path with the minimum number 
of IMPs was initially used. 12 This algorithm was later 
replaced by one which sends as much traffic as possible, 
along such paths until one or more circuits approach a 
few percent of full utilization. 28 These highly' utilized 
circuits arc then no longer allowed to carry additional 
flow. Instead, new paths with excess capacity and 
possibly more intermediate nodes are found. The 
procedure continues until some cutset contains only 
nearly fully utilized circuits. At this point no additional 
flow can be sent. For design purposes, this algorithm is 
a highly satisfactory replacement for the more com¬ 
plicated multi-commodity approach. Using the al¬ 
gorithm, it has been shown that the throughput capa¬ 
bilities of the. ARPA Network arc substantially 
insensitive to the distribution of traffic and depend 
mainly only on the total traffic flow within the network. 8 


In Reference 22, it was modified to describe the be¬ 
havior of ARPA-like computer networks while m 
Reference 23 it was refined further to apply directly to 
the ARPANET. 


The Binglc server model 


Queueing theory 20 provides an effective set of ana¬ 
lytical tools for studying packet delay. .Much of this 
theory considers systems in which messages place 
demands for transmission (service) upon a single, 
communication channel (the single server). These 
systems are characterized by A(t), the- distribution of 
interarrival times between demands and B(t), the 
distribution of service times. When the average demand 
for service is less than the capacity of the channel, the 
system is said to be stable. 

When A{t) is exponential (i.e., Poisson arrivals), 
and messages are transmitted on a first-come first-served 
basis, the average time T in the stable system is 


T = 


M 2 

2(1—p) 


-rf 


( 1 ) 


where X is the average arrival rate of messages, i and t 2 
are. the first and second moments of B{1) respectively, 
and p=X£<l. If the service time is also exponential, 


When A (r) and B(t) are arbitrary distributions, the 
situation becomes complex and only weak results are 
available. For example, no expression is available for T ; 
however the following upper bound yields an excellent 
approximation 19 as p—>1: 


T< 


A(<r„ 2 -poM) 

2(1—p) 


+ t 


( 3 ) 


where <r 0 2 and or are the variance of the interarrival 
time and service time distributions, respectively. 


Analytic models of network performance 

The effort to determine analytic models of system 
performance has proceeded in two phases: (I) the pre¬ 
diction of average time delay encountered by a message 
as it passes through the network, and (2) the use of 
these queueing models to calculate optimum channel 
capacity assignments for minimum possible delay. The 
model used as a standard for the average message delav 
was first described in Reference 21 where it served to 
predict delays in stochastic communication networks. 


Networks of queues 

Multiple channels in a network environment, give 
rise to queueing problems that are far more difficult to 
solve than single server systems. For example, the 
variability in the choice of source and destination for a 
message is a network phenomenon which contributes to 
delay. A principal analytical difficulty results from the 
fact that flows throughout the network are correlated. 
The basic approach to solving these stochastic network 
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problems iB to decompose them into nnalyzable single- 
server problems which reflect the original network 
structure and traffic flow. 

Early studies of queueing networks indicated that 
such a decomposition was possible however, those 
results do not carry over to message switched computer 
networks due to the correlation of traffic flows. In 
Reference 21 it was shown for a wide variety of com¬ 
munication nets that this correlation could be removed 
by considering the length of a given packet to be an 
independent random variable as it passes from node to 
node. Although this “independence” assumption is not 
physically realistic, it results in a mathematically 
tractable model which does not seem to affect the 
accuracy of the predicted time delays. As the size and 
connectivity of the network increases, the assumption 
becomes increasingly more realistic. With this assump¬ 
tion, a successful decomposition which permits a 
channel-by-channcl analysis is possible, as follows. 

The packet delay iB defined as the time which a 
pucKet Bpends in the network from its entry until it 
reaches its destination. The average packet delay is 
'knoted as T. Let Z,-* be the average delay for those 
packets whose origin is IMP j and whose destination is 
IMP k. We assume a Poisson arrival process for such 
packets with an average of yp packets per second and 
an exponential distribution of packet lengths with an 
average of 1/p bits per packet. With these definitions, 
if 7 is the sum of the quantities 7 ,•*, then 21 

T= 'Z — Zik (4) 

y 

Let us now reformulate Equation (4) in terms of 
single channel delays. We first define the following 
quantities for the ith channel: C, as its capacity 
(bits/second); X, as the average packet traffic it carries 
(packets/second); and 7\ as the average time a packet 
spends waiting for and using the ith channel. By 
relating the {X, 1 to the | 7 ,») via the paths selected by 
the routing algorithm, it is easy to see that 21 

T= Z-7\ (5) 

T 

With the assumption of Poisson traffic and exponential 
service times, the quantities 7\ are given by Equation 
(2). For an average packet length of 1/p bits, f = 1/pCi 
seconds and thus 


7 > 


1 


pCi —A,- 


( 6 ) 


Thus we have successfully decomposed the analysis 
problem into a set of simple singlc-ehanncl problems. 


A refinement of the. decomposition permits a non- 
exponential packet length distribution and uses Equa¬ 
tion ( 1 ) rather than Equation ( 2 ) to calculate 7\; 
as an approximation, the Markovian character of the 
traffic is assumed to be preserved. Furthermore, for 
computer networks we include the effect of propagation 
time and overhead traffic to obtain the following 
equation for average packet delay 22 ’ 22 


t=k+ Z- 

i 7 



x./V c, 

pCi — A, 


+P<+K 


(7) 


Here, 1/p' represents the average length of a Host 
packet, and 1 /p represents the average length of all 
packets (including acknowledgments, headers, requests 
for next messages, parity checks, etc.) within the net¬ 
work. The expression 1 / p' C i-\-{i/Ki/pC /) / {pC i— Xi)]+P,- 
represents the average packet delay on the ith channel. 
The term (\i/pC/) / (pC A,-) is the average time a 
packet spends waiting at the IMP for the ith channel to 
become available. Since the packet must compete with 
acknowledgments and other overhead traffic, the 
overall average packet length 1/p appears in the 
expression. The term 1/p'Ci is the time required to 
transmit a packet of average length p. Finally: K is 
the nodal processing time, assumed constant, and for 
the ARPA IMP approximately equal to 0.35 ms; 
Pi is the propagation time on the ith channel (about 
20 ms for a 3000 mile channel). 

Assuming a relatively homogeneous set of C, and 
Pi, no individual term in the expression for delay will 
dominate the summation until the flow A i/p in one 
channel (say channel i„) approaches the capacity C,„. 
At that point, the term 7\ 0 , and hence T will grow 
rapidly. The expression for delay is then dominated by 
one (or more) terms and exhibits a threshold behavior. 
Prior to this threshold, T remains relatively constant. 

The accuracy of the time delay model, as well as this 
threshold phenomenon was demonstrated on a 19 node 
network 1 * and on the ten node ARPA net derived from 
Figure 1(c) by deleting the rightmost five IMPs. 
Using the routing procedure described in the last 
section 28 and equal traffic bet ween all node pairs, the 
channel flows A, were found for the ten node net and 
the delay curves shown in Figure 3 were obtained. 
Curve A was obtained with fixed 1000 bit packets,* 
while curve D was generated for exponentially dis¬ 
tributed variable length packets with average size of 
500 bits. In both eases A and B, all overhead factors 
were ignored. Note that the delay remains small until a 


’ In case A, the application of Equation (1) allows for constant 
packet lengtlis (j.e., zero vuriance). 
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total throughput slightly greater than 400 kilobits/ 
second is reached. The delay then increases rapidly. 
Curves C and D respectively represent the same 
situations when the overhead of 136 bits per packet and 
per RFNM and 152 bits per acknowledgment are 
included. Notice that the total throughput per IMP 
is reduced to 250 kilobits/second in case C and to 
approximately 200 kilobits/socond in case D. 

In the same figure, wc have illustrated with t’s the 
results of a simulation performed with a realistic 
routing and metering strategy. The simulation omitted 
all network overhead and assumed fixed lengths of 
1000 bits for all packets. 

It is difficult to develop a practical routing and flow 
control procedure that will allow each IMP to input 
identical amounts of traffic. To compare the delay 


curve A with the points obtained by simulation, the 
curve Bhould actually be recomputed for the slightly 
skewed distribution that resulted. It is notable that the 
delay estimates from the simulation (which used a 
dynamic routing strategy) and the computation (which 
used a Btatic routing strategy and the time delay for¬ 
mula) arc in close agreement. In particular, they both 
accurately determined the. vertical rise of the delay 
curve in the range just above 400 kilobits/second, the 
formula by predicting infinite delay and the simulation 
by rejecting the further input of traffic. 

In practice and from the analytic and simulation 
studies of the ARPANET, the average queueing delay 
is observed to remain small (almost that, of an unloaded 
net) and well within the design constraint of 0.2 seconds 
until the traffic within the network approaches the 
capacity of a cutset. The delay then increases rapidly. 
Thus, as long as traffic is low enough and the routing 
adaptive enough to avoid the premature saturation of 
cutsets by guiding traffic along paths with excess capacity, 
queueing delays are not significant. 

Channel capacity optimization 

One of the most difficult design problems is the 
optimal selection of capacities from a finite set of 
options. Although there arc many heuristic approaches 
to this problem, analytic results arc relatively scarce. 
(For the specialized case of centralized networks, an 
algorithm yielding optimal results is available 1 . 11 ) While 
it is possible to find an economical assignment of 
discrete capacities for, say, a 200 IMP network, very 
little is known about the relation between such capacity 
assignments, message delay, and cost. 

To obtain theoretical properties of optimal capacity 
assignments, one may ignore the constraint that 
capacities arc obtainable only in discrete sizes. In 
Reference. 21 such a problem was posed where the 
network topology and average traffic flow were assumed 
to be known and fixed and an optimal match of capaci¬ 
ties to traffic flow was found. Also, the traffic was 
assumed to be Markovian (Poisson arrivals and 
exponential packet lengths) and the independence 
assumption and decomposition method were applied. 
For each channel, the capacity C, was found which 
minimized the average message delay T, at a fixed 
total system cost D. Since- X./g is the average bit rate on 
the fill channel, the solution to any optimal assignment 
problem must provide more than this minimal capacity 
to each channel. This is clear since both Equations (6) 
and (7) indicate that 7\ will become arbitrarily large 
with less than (or equal to) this amount of capacity. 

It is not critical exactly how the excess capacity is 
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assigned, as long as C,>X,/m. Other important param¬ 
eters and insights have been identified'in studying the 
continuous capacity optimization problem. For ex¬ 
ample, the number of excess dollars, D„ remaining 
after the minimum capacity A,/a is assigned to each 
channel is of great importance.. As D.—> 0, the average 
delay must grow arbitrarily large. In this range, the 
critical parameters become p and n where p = y/pC is 
the ratio of the rate y/p at which bits enter the network 
to the rate C at which the net can handle bits and 
n = X/ y, where X = ]TjA, is the total rate at which packets 
flow within the net. The quantity p represents a dimen¬ 
sionless form of network “load" whereas n is easily 
shown to represent the average path length for a 
packet. 

As the load p approaches 1/n, the delay T grows very 
quickly, and this point p- 1/n represents the maximum 
load which the network can support. If capacities arc 
assigned optimally, all channels saturate simultaneously 
at this point. In this formulation n is a design parameter 
which depends upon the topology and the routing 
procedure, while p is a parameter which depends upon 
the mput rate and the total capacity of the network. 
In studying the ARPANET 23 a closer representation 
of the actual tariffs for high speed telephone data 
channels used in that network was provided by setting 
D= ^idiCi where 0<a<l.* This approach requires 
the solution of a non-linear equation by numerical 
techniques. On solving the equation, it can be shown 
that the packet delay T varies insignificantly with a 
for .3<a<l. This indicates that the closed form 
solution discussed earlier with a=l is a reasonable 
approximation to the more difficult non-linear problem. 
These continuous capacity studies have application to 
general network studies (e.g., satellite communications)" 
and arc under continued investigation. 26 ' 2e " 

In practice, the selection of channel capacities must 
be made from a small finite set. Although some theo¬ 
retical work has been done in this case by approxi¬ 
mating the discrete cost-capacity functions by 
continuous ones, much remains to be done. 13 ' 24 Because 
of the discrete capacities and the time varying nature 
of network traffic, it is not generally possible to match 
channel capacities to the anticipated flows within the 
channels. If this were possible, all channels would 
saturate at the same externally applied load. Instead, 
capacities arc assigned on the basis of reasonable 
estimates of average or peak traffic flows. It is the 
responsibility of the. routing procedure to permit the 
traffic to adapt to the available capacity. 14 Often two 


• Of eourEc the tariffs reflect the discrete nature of available 
channels. The use of the exponent a provides a continuous fit 
to the discrete cost function. For the ARPANET, a=.8. 


IMP sites will engage in heavy communication and 
thus saturate one or more critical network cutsets. In 
such cases, the routing will not be able to send addi¬ 
tional flow across these cuts. The network will therefore 
experience “premature" saturation in one or a small set 
of channels leading to the threshold behavior described 
earlier. - . •- 

DISCUSSION 

A major conclusion from our experience in network 
design is that message switched networks of the ARPA 
type are no longer difficult to specify. They may be 
implemented straightforwardly from the specifications; 
they can be less expensive than other currently available 
technical approaches; they perform remarkably well as 
a communication system for interconnecting time¬ 
sharing and batch processing computers and can be 
adapted to directly handle teletypes, displays and many 
other kinds of terminal devices and data processing 
equipment. I# - M 

The principal tools available for the design of net¬ 
works are analysis, simulation, heuristic procedures, 
and experimentation. Analysis, simulation and heuristics 
have been the mainstays of the work on modeling and 
topological optimization while simulation, heuristic 
procedures and experimental techniques have been the 
major tools for the actual network implementation. 
Experience has shown that all of these methods are 
useful while none are all powerful. The most valuable 
approach has been the simultaneous use of several of 
these tools. 

Each approach has room for considerable improve¬ 
ment. The analysis efforts have not yet yielded results 
in many important areas such as routing. However, for 
prediction of delay, this approach leads to a simple 
threshold model which is both accurate and under¬ 
standable. Heuristic procedures all suffer from the 
problem that it is presently unclear how to select 
appropriate heuristics. It has been the innovative use 
of computers and analysis that has made the approach 
work well. For designing networks with no more than a 
few hundred IMPs, present heuristics appear adequate 
but a good deal of additional work is required for net¬ 
works of greater size. Simulation is a well developed tool 
that is both expensive to apply and limited in the overall 
understanding that it. yields. For these reasons, simula¬ 
tion appears to be most useful only in validating models, 
and in assisting in detailed design decisions such as the 
number of buffers that an IMP should contain. As the 
size of networks continues to grow, it appears that, 
simulation will become virtually useless as a total design 
tool. The ultimate standard by which all models and 
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conclusions cun be tested is experimentation. Experi¬ 
mentation with the actual network is conceptually 
relatively straightforward and very useful. Although, 
experiments are often logistically difficult, to perform, 
they can provide an easy means for testing models, 
heuristics and design parameters. 

The outstanding design problems currently facing 
the network designer are to specify and determine the 
properties of the routing, flow control and topological 
structure for large networks. This specification must 
rru ke full use, of a wide variety of circuit options. 
Preliminary studies indicate that initially, the most 
fruitful approaches will be based on the partitioning of 
the network into regioas, or equivalently, coastructing 
a large network by connecting a number of regional 
networks. To send a message, a Host would specify 
both the destination region and the destination IMP 
in that region. No detailed implementation of a large 
network has yet been specified but early studies of their 
properties indicate that factors such as cost, throughput, 
delay and reliability arc similar to those of the present 
ARPANET, if the ARPA technology is used.® 

Teen nques applicable to the design of large networks 
arc presently under intensive study. These techniques 
appear to split into the same four categories as small 
network design but approaches may differ significantly. 
For example, large nets arc likely to demand the place¬ 
ment of high bandwidth circuits at certain key locations 
in the topology to concentrate flow. These circuits will 
require the development of a high speed IMP to connect 
them into the net. It is likely that this high speed IMP 
will have the structure of a high speed multiplexor, and 
may require several cooperating processors to obtain 
the needed computer power for the job. Flow control 
strategics for large networks seem to extrapolate nicely 
from small network strategics if each region in the large 
network is viewed as a node in a smaller network. 
However, this area will require additional study as will 
the problem of specifying effective adaptive routing 
mechanisms. Recent efforts indicate that efficient, 
practical schemes for small networks will soon be 
available. These schemes seem to be applicable for 
adaptive routing and flow control in networks con¬ 
structed from regional subnetworks. The development 
of practical algorithms to handle routing and flow 
control is still an art rather than a science. Simulation is 
useful for studying the properties of a given heuristic, 
but intuition still plays a dominant role in the system 
design. 

Several open questions in network’ design presently 
are: (I) what structure should a high bandwidth IMP 
have; (2) how ear. full use be made of a variety of high, 
bandwidth circuits; (3) how should large networks be 
partitioned for both effective design and operation; 


find (4) what operational procedures should large 
networks follow? Much work has already been done in 
these areas but much more remains to be done. We 
expect substantial progress to be achieved in the next, 
few years, and accordingly, the increased understanding 
of the properties of message switched networks of 
all sizes. 
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INTRODUCTION 

Much has been said about the mechanics of the ARPA 
Computer Network (ARPANET) and especially about 
the organization of its communications subnet. 1 5 
Until recently the main effort has gone into the imple¬ 
mentation of an ARPANET user-level communications 
interface. Operating just above the communications 
subnet in ARPANET HOST Computers, this ARPA¬ 
NET interface is intended to serve as a foundation for 
the organization of function-oriented communica¬ 
tions.®' 7 See Figures I and 2 for our view of a computer 
system and the scheme for user-level process-to-process 
communications. It is now appropriate to review the 
development of protocols which have been constructed 
to promote particular substantive uses of the 
ARPANET, namely function-oriented protocols. 

We should begin this brief examination by stating 
what we mean by the word “protocol” and how proto¬ 
cols fit in the plan for useful work on the ARPANET. 
When we have two processes facing each other across 
some communication link, the protocol is the set of 
their agreements on the format and relative timing of 
messages to be exchanged. When we speak of a proto¬ 
col, there is usually an important goal to be fulfilled. 
Although any set of agreements between cooperating 
(i.e., communicating) processes is a protocol, the proto¬ 
cols of interest are those which arc constructed for 
general application by a large population of processes 
in solving a large class of problems. 

In the understanding and generation of protocols 
there arc two kinds of distinctions made. Protocols in 
the ARPANET arc layered and wc speak of high or 
low level protocols. High level protocols arc those most 
closely matched to functions and low level protocols 
deal with communications mechanics. The lowest level 
software protocols in the ARPANET involve reliable 


message exchange'- between ARPANET Interface 
Message Processors (IMPs). 2 -® A high level protocol is 
one with primitives closely related to a substantive use. 
At the lowest levels the contents of messages arc un¬ 
specified. At higher levels, more and more is stated 
about the meaning of message contents and timing. The 
layers of protocol are shown in Figure 3. 

A second way of structuring sets of protocols and 
their design is bound up in the word factoring. At any 
level of protocol are sets of format and timing rules 
associated with particular groupings of agreements. In 
the IMPs wc find certain protocols pertaining to error 
handling, while others to flow control, and still others 
to message routing. At the ARPANET’S user-level 
communications interface there arc, among others, 
separable protocols associated with cc*v-i;-u.i nr - "on- 
ncctions and logical data blocking. Tiieso piotocols do 
not nest, but join as modules at the same level. 

Before moving on to consider the higher level func¬ 
tion-oriented protocols, let us first make a few state¬ 
ments about underlying protocols. There arc three 
lower level software protocols which nest in support of 
the user-level communications interface for the ARPA¬ 
NET. The lowest of these is the IMP-IMP protocol 
which provides for reliable communication among 
IMPs. This protocol handles transmission-error detec¬ 
tion and correction, flow control to avoid message 
congestion, and routing. At the next higher level is the 
IMP-HOST protocol which provides for the passage 
of messages between HOSTs and IMPs in such a way 
as to create virtual communication paths between 
HOSTs. With IMP-HOST protocol, a HOST has 
operating rules which permit it to send messages to 
specified HOSTs on the ARPANET and to be informed 
of the dispensation of those messages. In particular, the 
IMP-HOST protocol constrains HOSTs in their trans¬ 
missions so that they can make good use of available 
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Figure 1—Our view of a computer system 


communications capacity without denying such avail¬ 
ability to other HOSTs. 

The HOST-HOST protocol, finally, is the set of 
rules whereby HOSTs construct and maintain com¬ 
munication between processes (user jobs) running on 
remote computers. One process requiring communica¬ 
tions with another on some remote computer system 
makes requests on its local supervisor to act on its 
behalf in establishing and maintaining those communi¬ 
cations under HOST-HOST protocol. 

In constructing these low levels of protocol it was the 
intention to provide user processes with a general set 
of useful communication primitives to isolate them 
from many of the details of operating systems and 
communications. At this user-level interface function- 
oriented protocols join as an open-ended collection of 
modules to make use of ARPANET capabilities. 

The communications environment facing the de¬ 
signers of function-oriented protocols in the ARPANET 


is essentially that of a system of one-way bvte-oriented 
connections. Technically speaking, a “connection” is a 
pair: a "send socket” at one end and a "receive socket” 
at the other. Primitives provided at the user-level 
interface include: 

1. Initiate connection (local socket., foreign socket), 

2. Wait for connection (local socket), 

3. Send, Receive (local socket, data), 

4. Close (local socket), 

5. Send interrupt signal (local socket). 

Processes in this virtual process network can create 
connections and transmit bytes. Connections arc sub¬ 
ject to HOST-HOST flow control and the vagaries of 
timing in a widely distributed computing environment, 
but care has been taken to give user processes control 
over their communications so as to make full use of 
network parallelism and redundancy. The kind of 
agreements which must be made in the creation of 
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Figure 2—Two communicating processes 


function-oriented protocols relate to rules for estab¬ 
lishing connections, to the timing rules which govern 
transmission sequences, and to the content of the byte- 
streams themselves. 

USE OF REMOTE INTERACTIVE SYSTEMS 

The application which currently dominates ARPA¬ 
NET activity is the remote use of interactive systems. 
A Telecommunications Network (TELNET) protocol 
is followed by processes cooperating to support this 
application.® A user at a terminal, connected to his 
local HOST, controls a process in a remote HOST as if 
he were a local user of the remote HOST. His local 
HOST copies characters between his terminal and 
TELNET connections over the ARPANET. Wc refer 
to the HOST where the user sits as the using HOST , 
and to the remote HOST as the serving HOST. Sec 
Figure 4. 


At the using HOST, the user must be able to per¬ 
form the following functions through his TELNET 
user process (“user-TELNET”): 

1. Initiate a pair of connections to a serving HOST, 

2. Send characters to the serving HOST, 

3. Receive characters from the serving HOST, 

4. Send a HOST-HOST interrupt signal, 

5. Terminate connections. 

The user-TELNET needs to be able to distinguish be¬ 
tween (1) commands to be acted on locally and (2) 
input intended for the serving HOST. An escape char¬ 
acter is reserved to mark local commands. Conventions 
for the ARPANET Terminal IMP (TIP) user- 
TELNET are typical.' 

In most using HOSTs, the above functions arc pro¬ 
vided by a user-TELNET which is a user-level ■program. 
A minimal user-TELNET need only implement the 
above functions, but several additional support func- 



